Merge lp:~blr/charms/precise/squid-forwardproxy/fix-unit-state into lp:~canonical-launchpad-branches/charms/precise/squid-forwardproxy/trunk

Proposed by Kit Randel on 2015-12-15
Status: Merged
Merged at revision: 35
Proposed branch: lp:~blr/charms/precise/squid-forwardproxy/fix-unit-state
Merge into: lp:~canonical-launchpad-branches/charms/precise/squid-forwardproxy/trunk
Diff against target: 3974 lines (+3760/-43)
20 files modified
Makefile (+13/-0)
charm-helpers.yaml (+4/-0)
hooks/charmhelpers/__init__.py (+38/-0)
hooks/charmhelpers/core/__init__.py (+15/-0)
hooks/charmhelpers/core/decorators.py (+57/-0)
hooks/charmhelpers/core/files.py (+45/-0)
hooks/charmhelpers/core/fstab.py (+134/-0)
hooks/charmhelpers/core/hookenv.py (+978/-0)
hooks/charmhelpers/core/host.py (+641/-0)
hooks/charmhelpers/core/hugepage.py (+71/-0)
hooks/charmhelpers/core/kernel.py (+68/-0)
hooks/charmhelpers/core/services/__init__.py (+18/-0)
hooks/charmhelpers/core/services/base.py (+353/-0)
hooks/charmhelpers/core/services/helpers.py (+292/-0)
hooks/charmhelpers/core/strutils.py (+72/-0)
hooks/charmhelpers/core/sysctl.py (+56/-0)
hooks/charmhelpers/core/templating.py (+81/-0)
hooks/charmhelpers/core/unitdata.py (+521/-0)
hooks/hooks.py (+50/-43)
scripts/charm_helpers_sync.py (+253/-0)
To merge this branch: bzr merge lp:~blr/charms/precise/squid-forwardproxy/fix-unit-state
Reviewer Review Type Date Requested Status
William Grant code 2015-12-15 Approve on 2015-12-17
Review via email: mp+280650@code.launchpad.net

Commit message

Manage local unit state with charmhelpers unitdata.kv.

Description of the change

This branch correctly manages the persistence of hook states 'start' and 'relation-auth-helper-joined' using the charmhelpers kv store, instead of the broken and nonsensical mess I merged earlier. Mea culpa.

To post a comment you must log in.
William Grant (wgrant) :
review: Needs Fixing (code)
63. By Kit Randel on 2015-12-16

Unset hook_start unitdata on stop hook.

64. By Kit Randel on 2015-12-16

Reloading squid3 is redundant given fallthrough restart case.

65. By Kit Randel on 2015-12-17

Remove unnecessary unitdata keys 'state_delayed_start' and 'hook_auth_helper_joined'.

66. By Kit Randel on 2015-12-17

* Remove potential vector for symlink attack.
* Update charmhelpers.

67. By Kit Randel on 2015-12-17

Factor out test for delayed service start.

68. By Kit Randel on 2015-12-17

Move invalid squid configuration CRITICAL log to service_squid3('check').

William Grant (wgrant) :
review: Approve (code)
69. By Kit Randel on 2015-12-17

Unset hook_start unitdata key, regardless of service status.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'Makefile'
2--- Makefile 1970-01-01 00:00:00 +0000
3+++ Makefile 2015-12-17 05:44:32 +0000
4@@ -0,0 +1,13 @@
5+# -*- mode: makefile -*-
6+
7+PYTHON := /usr/bin/env python
8+
9+all: sync
10+
11+sync:
12+ @bzr cat lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py > scripts/charm_helpers_sync.py
13+ @echo "Syncing charmhelpers..."
14+ @mkdir -p hooks/charmhelpers
15+ @$(PYTHON) scripts/charm_helpers_sync.py -c charm-helpers.yaml
16+
17+
18
19=== added file 'charm-helpers.yaml'
20--- charm-helpers.yaml 1970-01-01 00:00:00 +0000
21+++ charm-helpers.yaml 2015-12-17 05:44:32 +0000
22@@ -0,0 +1,4 @@
23+destination: hooks/charmhelpers
24+branch: lp:charm-helpers
25+include:
26+ - core
27\ No newline at end of file
28
29=== added directory 'hooks/charmhelpers'
30=== added file 'hooks/charmhelpers/__init__.py'
31--- hooks/charmhelpers/__init__.py 1970-01-01 00:00:00 +0000
32+++ hooks/charmhelpers/__init__.py 2015-12-17 05:44:32 +0000
33@@ -0,0 +1,38 @@
34+# Copyright 2014-2015 Canonical Limited.
35+#
36+# This file is part of charm-helpers.
37+#
38+# charm-helpers is free software: you can redistribute it and/or modify
39+# it under the terms of the GNU Lesser General Public License version 3 as
40+# published by the Free Software Foundation.
41+#
42+# charm-helpers is distributed in the hope that it will be useful,
43+# but WITHOUT ANY WARRANTY; without even the implied warranty of
44+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
45+# GNU Lesser General Public License for more details.
46+#
47+# You should have received a copy of the GNU Lesser General Public License
48+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
49+
50+# Bootstrap charm-helpers, installing its dependencies if necessary using
51+# only standard libraries.
52+import subprocess
53+import sys
54+
55+try:
56+ import six # flake8: noqa
57+except ImportError:
58+ if sys.version_info.major == 2:
59+ subprocess.check_call(['apt-get', 'install', '-y', 'python-six'])
60+ else:
61+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-six'])
62+ import six # flake8: noqa
63+
64+try:
65+ import yaml # flake8: noqa
66+except ImportError:
67+ if sys.version_info.major == 2:
68+ subprocess.check_call(['apt-get', 'install', '-y', 'python-yaml'])
69+ else:
70+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
71+ import yaml # flake8: noqa
72
73=== added directory 'hooks/charmhelpers/core'
74=== added file 'hooks/charmhelpers/core/__init__.py'
75--- hooks/charmhelpers/core/__init__.py 1970-01-01 00:00:00 +0000
76+++ hooks/charmhelpers/core/__init__.py 2015-12-17 05:44:32 +0000
77@@ -0,0 +1,15 @@
78+# Copyright 2014-2015 Canonical Limited.
79+#
80+# This file is part of charm-helpers.
81+#
82+# charm-helpers is free software: you can redistribute it and/or modify
83+# it under the terms of the GNU Lesser General Public License version 3 as
84+# published by the Free Software Foundation.
85+#
86+# charm-helpers is distributed in the hope that it will be useful,
87+# but WITHOUT ANY WARRANTY; without even the implied warranty of
88+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
89+# GNU Lesser General Public License for more details.
90+#
91+# You should have received a copy of the GNU Lesser General Public License
92+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
93
94=== added file 'hooks/charmhelpers/core/decorators.py'
95--- hooks/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
96+++ hooks/charmhelpers/core/decorators.py 2015-12-17 05:44:32 +0000
97@@ -0,0 +1,57 @@
98+# Copyright 2014-2015 Canonical Limited.
99+#
100+# This file is part of charm-helpers.
101+#
102+# charm-helpers is free software: you can redistribute it and/or modify
103+# it under the terms of the GNU Lesser General Public License version 3 as
104+# published by the Free Software Foundation.
105+#
106+# charm-helpers is distributed in the hope that it will be useful,
107+# but WITHOUT ANY WARRANTY; without even the implied warranty of
108+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
109+# GNU Lesser General Public License for more details.
110+#
111+# You should have received a copy of the GNU Lesser General Public License
112+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
113+
114+#
115+# Copyright 2014 Canonical Ltd.
116+#
117+# Authors:
118+# Edward Hope-Morley <opentastic@gmail.com>
119+#
120+
121+import time
122+
123+from charmhelpers.core.hookenv import (
124+ log,
125+ INFO,
126+)
127+
128+
129+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
130+ """If the decorated function raises exception exc_type, allow num_retries
131+ retry attempts before raise the exception.
132+ """
133+ def _retry_on_exception_inner_1(f):
134+ def _retry_on_exception_inner_2(*args, **kwargs):
135+ retries = num_retries
136+ multiplier = 1
137+ while True:
138+ try:
139+ return f(*args, **kwargs)
140+ except exc_type:
141+ if not retries:
142+ raise
143+
144+ delay = base_delay * multiplier
145+ multiplier += 1
146+ log("Retrying '%s' %d more times (delay=%s)" %
147+ (f.__name__, retries, delay), level=INFO)
148+ retries -= 1
149+ if delay:
150+ time.sleep(delay)
151+
152+ return _retry_on_exception_inner_2
153+
154+ return _retry_on_exception_inner_1
155
156=== added file 'hooks/charmhelpers/core/files.py'
157--- hooks/charmhelpers/core/files.py 1970-01-01 00:00:00 +0000
158+++ hooks/charmhelpers/core/files.py 2015-12-17 05:44:32 +0000
159@@ -0,0 +1,45 @@
160+#!/usr/bin/env python
161+# -*- coding: utf-8 -*-
162+
163+# Copyright 2014-2015 Canonical Limited.
164+#
165+# This file is part of charm-helpers.
166+#
167+# charm-helpers is free software: you can redistribute it and/or modify
168+# it under the terms of the GNU Lesser General Public License version 3 as
169+# published by the Free Software Foundation.
170+#
171+# charm-helpers is distributed in the hope that it will be useful,
172+# but WITHOUT ANY WARRANTY; without even the implied warranty of
173+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
174+# GNU Lesser General Public License for more details.
175+#
176+# You should have received a copy of the GNU Lesser General Public License
177+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
178+
179+__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'
180+
181+import os
182+import subprocess
183+
184+
185+def sed(filename, before, after, flags='g'):
186+ """
187+ Search and replaces the given pattern on filename.
188+
189+ :param filename: relative or absolute file path.
190+ :param before: expression to be replaced (see 'man sed')
191+ :param after: expression to replace with (see 'man sed')
192+ :param flags: sed-compatible regex flags in example, to make
193+ the search and replace case insensitive, specify ``flags="i"``.
194+ The ``g`` flag is always specified regardless, so you do not
195+ need to remember to include it when overriding this parameter.
196+ :returns: If the sed command exit code was zero then return,
197+ otherwise raise CalledProcessError.
198+ """
199+ expression = r's/{0}/{1}/{2}'.format(before,
200+ after, flags)
201+
202+ return subprocess.check_call(["sed", "-i", "-r", "-e",
203+ expression,
204+ os.path.expanduser(filename)])
205
206=== added file 'hooks/charmhelpers/core/fstab.py'
207--- hooks/charmhelpers/core/fstab.py 1970-01-01 00:00:00 +0000
208+++ hooks/charmhelpers/core/fstab.py 2015-12-17 05:44:32 +0000
209@@ -0,0 +1,134 @@
210+#!/usr/bin/env python
211+# -*- coding: utf-8 -*-
212+
213+# Copyright 2014-2015 Canonical Limited.
214+#
215+# This file is part of charm-helpers.
216+#
217+# charm-helpers is free software: you can redistribute it and/or modify
218+# it under the terms of the GNU Lesser General Public License version 3 as
219+# published by the Free Software Foundation.
220+#
221+# charm-helpers is distributed in the hope that it will be useful,
222+# but WITHOUT ANY WARRANTY; without even the implied warranty of
223+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
224+# GNU Lesser General Public License for more details.
225+#
226+# You should have received a copy of the GNU Lesser General Public License
227+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
228+
229+import io
230+import os
231+
232+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
233+
234+
235+class Fstab(io.FileIO):
236+ """This class extends file in order to implement a file reader/writer
237+ for file `/etc/fstab`
238+ """
239+
240+ class Entry(object):
241+ """Entry class represents a non-comment line on the `/etc/fstab` file
242+ """
243+ def __init__(self, device, mountpoint, filesystem,
244+ options, d=0, p=0):
245+ self.device = device
246+ self.mountpoint = mountpoint
247+ self.filesystem = filesystem
248+
249+ if not options:
250+ options = "defaults"
251+
252+ self.options = options
253+ self.d = int(d)
254+ self.p = int(p)
255+
256+ def __eq__(self, o):
257+ return str(self) == str(o)
258+
259+ def __str__(self):
260+ return "{} {} {} {} {} {}".format(self.device,
261+ self.mountpoint,
262+ self.filesystem,
263+ self.options,
264+ self.d,
265+ self.p)
266+
267+ DEFAULT_PATH = os.path.join(os.path.sep, 'etc', 'fstab')
268+
269+ def __init__(self, path=None):
270+ if path:
271+ self._path = path
272+ else:
273+ self._path = self.DEFAULT_PATH
274+ super(Fstab, self).__init__(self._path, 'rb+')
275+
276+ def _hydrate_entry(self, line):
277+ # NOTE: use split with no arguments to split on any
278+ # whitespace including tabs
279+ return Fstab.Entry(*filter(
280+ lambda x: x not in ('', None),
281+ line.strip("\n").split()))
282+
283+ @property
284+ def entries(self):
285+ self.seek(0)
286+ for line in self.readlines():
287+ line = line.decode('us-ascii')
288+ try:
289+ if line.strip() and not line.strip().startswith("#"):
290+ yield self._hydrate_entry(line)
291+ except ValueError:
292+ pass
293+
294+ def get_entry_by_attr(self, attr, value):
295+ for entry in self.entries:
296+ e_attr = getattr(entry, attr)
297+ if e_attr == value:
298+ return entry
299+ return None
300+
301+ def add_entry(self, entry):
302+ if self.get_entry_by_attr('device', entry.device):
303+ return False
304+
305+ self.write((str(entry) + '\n').encode('us-ascii'))
306+ self.truncate()
307+ return entry
308+
309+ def remove_entry(self, entry):
310+ self.seek(0)
311+
312+ lines = [l.decode('us-ascii') for l in self.readlines()]
313+
314+ found = False
315+ for index, line in enumerate(lines):
316+ if line.strip() and not line.strip().startswith("#"):
317+ if self._hydrate_entry(line) == entry:
318+ found = True
319+ break
320+
321+ if not found:
322+ return False
323+
324+ lines.remove(line)
325+
326+ self.seek(0)
327+ self.write(''.join(lines).encode('us-ascii'))
328+ self.truncate()
329+ return True
330+
331+ @classmethod
332+ def remove_by_mountpoint(cls, mountpoint, path=None):
333+ fstab = cls(path=path)
334+ entry = fstab.get_entry_by_attr('mountpoint', mountpoint)
335+ if entry:
336+ return fstab.remove_entry(entry)
337+ return False
338+
339+ @classmethod
340+ def add(cls, device, mountpoint, filesystem, options=None, path=None):
341+ return cls(path=path).add_entry(Fstab.Entry(device,
342+ mountpoint, filesystem,
343+ options=options))
344
345=== added file 'hooks/charmhelpers/core/hookenv.py'
346--- hooks/charmhelpers/core/hookenv.py 1970-01-01 00:00:00 +0000
347+++ hooks/charmhelpers/core/hookenv.py 2015-12-17 05:44:32 +0000
348@@ -0,0 +1,978 @@
349+# Copyright 2014-2015 Canonical Limited.
350+#
351+# This file is part of charm-helpers.
352+#
353+# charm-helpers is free software: you can redistribute it and/or modify
354+# it under the terms of the GNU Lesser General Public License version 3 as
355+# published by the Free Software Foundation.
356+#
357+# charm-helpers is distributed in the hope that it will be useful,
358+# but WITHOUT ANY WARRANTY; without even the implied warranty of
359+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
360+# GNU Lesser General Public License for more details.
361+#
362+# You should have received a copy of the GNU Lesser General Public License
363+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
364+
365+"Interactions with the Juju environment"
366+# Copyright 2013 Canonical Ltd.
367+#
368+# Authors:
369+# Charm Helpers Developers <juju@lists.ubuntu.com>
370+
371+from __future__ import print_function
372+import copy
373+from distutils.version import LooseVersion
374+from functools import wraps
375+import glob
376+import os
377+import json
378+import yaml
379+import subprocess
380+import sys
381+import errno
382+import tempfile
383+from subprocess import CalledProcessError
384+
385+import six
386+if not six.PY3:
387+ from UserDict import UserDict
388+else:
389+ from collections import UserDict
390+
391+CRITICAL = "CRITICAL"
392+ERROR = "ERROR"
393+WARNING = "WARNING"
394+INFO = "INFO"
395+DEBUG = "DEBUG"
396+MARKER = object()
397+
398+cache = {}
399+
400+
401+def cached(func):
402+ """Cache return values for multiple executions of func + args
403+
404+ For example::
405+
406+ @cached
407+ def unit_get(attribute):
408+ pass
409+
410+ unit_get('test')
411+
412+ will cache the result of unit_get + 'test' for future calls.
413+ """
414+ @wraps(func)
415+ def wrapper(*args, **kwargs):
416+ global cache
417+ key = str((func, args, kwargs))
418+ try:
419+ return cache[key]
420+ except KeyError:
421+ pass # Drop out of the exception handler scope.
422+ res = func(*args, **kwargs)
423+ cache[key] = res
424+ return res
425+ wrapper._wrapped = func
426+ return wrapper
427+
428+
429+def flush(key):
430+ """Flushes any entries from function cache where the
431+ key is found in the function+args """
432+ flush_list = []
433+ for item in cache:
434+ if key in item:
435+ flush_list.append(item)
436+ for item in flush_list:
437+ del cache[item]
438+
439+
440+def log(message, level=None):
441+ """Write a message to the juju log"""
442+ command = ['juju-log']
443+ if level:
444+ command += ['-l', level]
445+ if not isinstance(message, six.string_types):
446+ message = repr(message)
447+ command += [message]
448+ # Missing juju-log should not cause failures in unit tests
449+ # Send log output to stderr
450+ try:
451+ subprocess.call(command)
452+ except OSError as e:
453+ if e.errno == errno.ENOENT:
454+ if level:
455+ message = "{}: {}".format(level, message)
456+ message = "juju-log: {}".format(message)
457+ print(message, file=sys.stderr)
458+ else:
459+ raise
460+
461+
462+class Serializable(UserDict):
463+ """Wrapper, an object that can be serialized to yaml or json"""
464+
465+ def __init__(self, obj):
466+ # wrap the object
467+ UserDict.__init__(self)
468+ self.data = obj
469+
470+ def __getattr__(self, attr):
471+ # See if this object has attribute.
472+ if attr in ("json", "yaml", "data"):
473+ return self.__dict__[attr]
474+ # Check for attribute in wrapped object.
475+ got = getattr(self.data, attr, MARKER)
476+ if got is not MARKER:
477+ return got
478+ # Proxy to the wrapped object via dict interface.
479+ try:
480+ return self.data[attr]
481+ except KeyError:
482+ raise AttributeError(attr)
483+
484+ def __getstate__(self):
485+ # Pickle as a standard dictionary.
486+ return self.data
487+
488+ def __setstate__(self, state):
489+ # Unpickle into our wrapper.
490+ self.data = state
491+
492+ def json(self):
493+ """Serialize the object to json"""
494+ return json.dumps(self.data)
495+
496+ def yaml(self):
497+ """Serialize the object to yaml"""
498+ return yaml.dump(self.data)
499+
500+
501+def execution_environment():
502+ """A convenient bundling of the current execution context"""
503+ context = {}
504+ context['conf'] = config()
505+ if relation_id():
506+ context['reltype'] = relation_type()
507+ context['relid'] = relation_id()
508+ context['rel'] = relation_get()
509+ context['unit'] = local_unit()
510+ context['rels'] = relations()
511+ context['env'] = os.environ
512+ return context
513+
514+
515+def in_relation_hook():
516+ """Determine whether we're running in a relation hook"""
517+ return 'JUJU_RELATION' in os.environ
518+
519+
520+def relation_type():
521+ """The scope for the current relation hook"""
522+ return os.environ.get('JUJU_RELATION', None)
523+
524+
525+@cached
526+def relation_id(relation_name=None, service_or_unit=None):
527+ """The relation ID for the current or a specified relation"""
528+ if not relation_name and not service_or_unit:
529+ return os.environ.get('JUJU_RELATION_ID', None)
530+ elif relation_name and service_or_unit:
531+ service_name = service_or_unit.split('/')[0]
532+ for relid in relation_ids(relation_name):
533+ remote_service = remote_service_name(relid)
534+ if remote_service == service_name:
535+ return relid
536+ else:
537+ raise ValueError('Must specify neither or both of relation_name and service_or_unit')
538+
539+
540+def local_unit():
541+ """Local unit ID"""
542+ return os.environ['JUJU_UNIT_NAME']
543+
544+
545+def remote_unit():
546+ """The remote unit for the current relation hook"""
547+ return os.environ.get('JUJU_REMOTE_UNIT', None)
548+
549+
550+def service_name():
551+ """The name service group this unit belongs to"""
552+ return local_unit().split('/')[0]
553+
554+
555+@cached
556+def remote_service_name(relid=None):
557+ """The remote service name for a given relation-id (or the current relation)"""
558+ if relid is None:
559+ unit = remote_unit()
560+ else:
561+ units = related_units(relid)
562+ unit = units[0] if units else None
563+ return unit.split('/')[0] if unit else None
564+
565+
566+def hook_name():
567+ """The name of the currently executing hook"""
568+ return os.environ.get('JUJU_HOOK_NAME', os.path.basename(sys.argv[0]))
569+
570+
571+class Config(dict):
572+ """A dictionary representation of the charm's config.yaml, with some
573+ extra features:
574+
575+ - See which values in the dictionary have changed since the previous hook.
576+ - For values that have changed, see what the previous value was.
577+ - Store arbitrary data for use in a later hook.
578+
579+ NOTE: Do not instantiate this object directly - instead call
580+ ``hookenv.config()``, which will return an instance of :class:`Config`.
581+
582+ Example usage::
583+
584+ >>> # inside a hook
585+ >>> from charmhelpers.core import hookenv
586+ >>> config = hookenv.config()
587+ >>> config['foo']
588+ 'bar'
589+ >>> # store a new key/value for later use
590+ >>> config['mykey'] = 'myval'
591+
592+
593+ >>> # user runs `juju set mycharm foo=baz`
594+ >>> # now we're inside subsequent config-changed hook
595+ >>> config = hookenv.config()
596+ >>> config['foo']
597+ 'baz'
598+ >>> # test to see if this val has changed since last hook
599+ >>> config.changed('foo')
600+ True
601+ >>> # what was the previous value?
602+ >>> config.previous('foo')
603+ 'bar'
604+ >>> # keys/values that we add are preserved across hooks
605+ >>> config['mykey']
606+ 'myval'
607+
608+ """
609+ CONFIG_FILE_NAME = '.juju-persistent-config'
610+
611+ def __init__(self, *args, **kw):
612+ super(Config, self).__init__(*args, **kw)
613+ self.implicit_save = True
614+ self._prev_dict = None
615+ self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
616+ if os.path.exists(self.path):
617+ self.load_previous()
618+ atexit(self._implicit_save)
619+
620+ def load_previous(self, path=None):
621+ """Load previous copy of config from disk.
622+
623+ In normal usage you don't need to call this method directly - it
624+ is called automatically at object initialization.
625+
626+ :param path:
627+
628+ File path from which to load the previous config. If `None`,
629+ config is loaded from the default location. If `path` is
630+ specified, subsequent `save()` calls will write to the same
631+ path.
632+
633+ """
634+ self.path = path or self.path
635+ with open(self.path) as f:
636+ self._prev_dict = json.load(f)
637+ for k, v in copy.deepcopy(self._prev_dict).items():
638+ if k not in self:
639+ self[k] = v
640+
641+ def changed(self, key):
642+ """Return True if the current value for this key is different from
643+ the previous value.
644+
645+ """
646+ if self._prev_dict is None:
647+ return True
648+ return self.previous(key) != self.get(key)
649+
650+ def previous(self, key):
651+ """Return previous value for this key, or None if there
652+ is no previous value.
653+
654+ """
655+ if self._prev_dict:
656+ return self._prev_dict.get(key)
657+ return None
658+
659+ def save(self):
660+ """Save this config to disk.
661+
662+ If the charm is using the :mod:`Services Framework <services.base>`
663+ or :meth:'@hook <Hooks.hook>' decorator, this
664+ is called automatically at the end of successful hook execution.
665+ Otherwise, it should be called directly by user code.
666+
667+ To disable automatic saves, set ``implicit_save=False`` on this
668+ instance.
669+
670+ """
671+ with open(self.path, 'w') as f:
672+ json.dump(self, f)
673+
674+ def _implicit_save(self):
675+ if self.implicit_save:
676+ self.save()
677+
678+
679+@cached
680+def config(scope=None):
681+ """Juju charm configuration"""
682+ config_cmd_line = ['config-get']
683+ if scope is not None:
684+ config_cmd_line.append(scope)
685+ config_cmd_line.append('--format=json')
686+ try:
687+ config_data = json.loads(
688+ subprocess.check_output(config_cmd_line).decode('UTF-8'))
689+ if scope is not None:
690+ return config_data
691+ return Config(config_data)
692+ except ValueError:
693+ return None
694+
695+
696+@cached
697+def relation_get(attribute=None, unit=None, rid=None):
698+ """Get relation information"""
699+ _args = ['relation-get', '--format=json']
700+ if rid:
701+ _args.append('-r')
702+ _args.append(rid)
703+ _args.append(attribute or '-')
704+ if unit:
705+ _args.append(unit)
706+ try:
707+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
708+ except ValueError:
709+ return None
710+ except CalledProcessError as e:
711+ if e.returncode == 2:
712+ return None
713+ raise
714+
715+
716+def relation_set(relation_id=None, relation_settings=None, **kwargs):
717+ """Set relation information for the current unit"""
718+ relation_settings = relation_settings if relation_settings else {}
719+ relation_cmd_line = ['relation-set']
720+ accepts_file = "--file" in subprocess.check_output(
721+ relation_cmd_line + ["--help"], universal_newlines=True)
722+ if relation_id is not None:
723+ relation_cmd_line.extend(('-r', relation_id))
724+ settings = relation_settings.copy()
725+ settings.update(kwargs)
726+ for key, value in settings.items():
727+ # Force value to be a string: it always should, but some call
728+ # sites pass in things like dicts or numbers.
729+ if value is not None:
730+ settings[key] = "{}".format(value)
731+ if accepts_file:
732+ # --file was introduced in Juju 1.23.2. Use it by default if
733+ # available, since otherwise we'll break if the relation data is
734+ # too big. Ideally we should tell relation-set to read the data from
735+ # stdin, but that feature is broken in 1.23.2: Bug #1454678.
736+ with tempfile.NamedTemporaryFile(delete=False) as settings_file:
737+ settings_file.write(yaml.safe_dump(settings).encode("utf-8"))
738+ subprocess.check_call(
739+ relation_cmd_line + ["--file", settings_file.name])
740+ os.remove(settings_file.name)
741+ else:
742+ for key, value in settings.items():
743+ if value is None:
744+ relation_cmd_line.append('{}='.format(key))
745+ else:
746+ relation_cmd_line.append('{}={}'.format(key, value))
747+ subprocess.check_call(relation_cmd_line)
748+ # Flush cache of any relation-gets for local unit
749+ flush(local_unit())
750+
751+
752+def relation_clear(r_id=None):
753+ ''' Clears any relation data already set on relation r_id '''
754+ settings = relation_get(rid=r_id,
755+ unit=local_unit())
756+ for setting in settings:
757+ if setting not in ['public-address', 'private-address']:
758+ settings[setting] = None
759+ relation_set(relation_id=r_id,
760+ **settings)
761+
762+
763+@cached
764+def relation_ids(reltype=None):
765+ """A list of relation_ids"""
766+ reltype = reltype or relation_type()
767+ relid_cmd_line = ['relation-ids', '--format=json']
768+ if reltype is not None:
769+ relid_cmd_line.append(reltype)
770+ return json.loads(
771+ subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
772+ return []
773+
774+
775+@cached
776+def related_units(relid=None):
777+ """A list of related units"""
778+ relid = relid or relation_id()
779+ units_cmd_line = ['relation-list', '--format=json']
780+ if relid is not None:
781+ units_cmd_line.extend(('-r', relid))
782+ return json.loads(
783+ subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
784+
785+
786+@cached
787+def relation_for_unit(unit=None, rid=None):
788+ """Get the json represenation of a unit's relation"""
789+ unit = unit or remote_unit()
790+ relation = relation_get(unit=unit, rid=rid)
791+ for key in relation:
792+ if key.endswith('-list'):
793+ relation[key] = relation[key].split()
794+ relation['__unit__'] = unit
795+ return relation
796+
797+
798+@cached
799+def relations_for_id(relid=None):
800+ """Get relations of a specific relation ID"""
801+ relation_data = []
802+ relid = relid or relation_ids()
803+ for unit in related_units(relid):
804+ unit_data = relation_for_unit(unit, relid)
805+ unit_data['__relid__'] = relid
806+ relation_data.append(unit_data)
807+ return relation_data
808+
809+
810+@cached
811+def relations_of_type(reltype=None):
812+ """Get relations of a specific type"""
813+ relation_data = []
814+ reltype = reltype or relation_type()
815+ for relid in relation_ids(reltype):
816+ for relation in relations_for_id(relid):
817+ relation['__relid__'] = relid
818+ relation_data.append(relation)
819+ return relation_data
820+
821+
822+@cached
823+def metadata():
824+ """Get the current charm metadata.yaml contents as a python object"""
825+ with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
826+ return yaml.safe_load(md)
827+
828+
829+@cached
830+def relation_types():
831+ """Get a list of relation types supported by this charm"""
832+ rel_types = []
833+ md = metadata()
834+ for key in ('provides', 'requires', 'peers'):
835+ section = md.get(key)
836+ if section:
837+ rel_types.extend(section.keys())
838+ return rel_types
839+
840+
841+@cached
842+def peer_relation_id():
843+ '''Get the peers relation id if a peers relation has been joined, else None.'''
844+ md = metadata()
845+ section = md.get('peers')
846+ if section:
847+ for key in section:
848+ relids = relation_ids(key)
849+ if relids:
850+ return relids[0]
851+ return None
852+
853+
854+@cached
855+def relation_to_interface(relation_name):
856+ """
857+ Given the name of a relation, return the interface that relation uses.
858+
859+ :returns: The interface name, or ``None``.
860+ """
861+ return relation_to_role_and_interface(relation_name)[1]
862+
863+
864+@cached
865+def relation_to_role_and_interface(relation_name):
866+ """
867+ Given the name of a relation, return the role and the name of the interface
868+ that relation uses (where role is one of ``provides``, ``requires``, or ``peers``).
869+
870+ :returns: A tuple containing ``(role, interface)``, or ``(None, None)``.
871+ """
872+ _metadata = metadata()
873+ for role in ('provides', 'requires', 'peers'):
874+ interface = _metadata.get(role, {}).get(relation_name, {}).get('interface')
875+ if interface:
876+ return role, interface
877+ return None, None
878+
879+
880+@cached
881+def role_and_interface_to_relations(role, interface_name):
882+ """
883+ Given a role and interface name, return a list of relation names for the
884+ current charm that use that interface under that role (where role is one
885+ of ``provides``, ``requires``, or ``peers``).
886+
887+ :returns: A list of relation names.
888+ """
889+ _metadata = metadata()
890+ results = []
891+ for relation_name, relation in _metadata.get(role, {}).items():
892+ if relation['interface'] == interface_name:
893+ results.append(relation_name)
894+ return results
895+
896+
897+@cached
898+def interface_to_relations(interface_name):
899+ """
900+ Given an interface, return a list of relation names for the current
901+ charm that use that interface.
902+
903+ :returns: A list of relation names.
904+ """
905+ results = []
906+ for role in ('provides', 'requires', 'peers'):
907+ results.extend(role_and_interface_to_relations(role, interface_name))
908+ return results
909+
910+
911+@cached
912+def charm_name():
913+ """Get the name of the current charm as is specified on metadata.yaml"""
914+ return metadata().get('name')
915+
916+
917+@cached
918+def relations():
919+ """Get a nested dictionary of relation data for all related units"""
920+ rels = {}
921+ for reltype in relation_types():
922+ relids = {}
923+ for relid in relation_ids(reltype):
924+ units = {local_unit(): relation_get(unit=local_unit(), rid=relid)}
925+ for unit in related_units(relid):
926+ reldata = relation_get(unit=unit, rid=relid)
927+ units[unit] = reldata
928+ relids[relid] = units
929+ rels[reltype] = relids
930+ return rels
931+
932+
933+@cached
934+def is_relation_made(relation, keys='private-address'):
935+ '''
936+ Determine whether a relation is established by checking for
937+ presence of key(s). If a list of keys is provided, they
938+ must all be present for the relation to be identified as made
939+ '''
940+ if isinstance(keys, str):
941+ keys = [keys]
942+ for r_id in relation_ids(relation):
943+ for unit in related_units(r_id):
944+ context = {}
945+ for k in keys:
946+ context[k] = relation_get(k, rid=r_id,
947+ unit=unit)
948+ if None not in context.values():
949+ return True
950+ return False
951+
952+
953+def open_port(port, protocol="TCP"):
954+ """Open a service network port"""
955+ _args = ['open-port']
956+ _args.append('{}/{}'.format(port, protocol))
957+ subprocess.check_call(_args)
958+
959+
960+def close_port(port, protocol="TCP"):
961+ """Close a service network port"""
962+ _args = ['close-port']
963+ _args.append('{}/{}'.format(port, protocol))
964+ subprocess.check_call(_args)
965+
966+
967+@cached
968+def unit_get(attribute):
969+ """Get the unit ID for the remote unit"""
970+ _args = ['unit-get', '--format=json', attribute]
971+ try:
972+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
973+ except ValueError:
974+ return None
975+
976+
977+def unit_public_ip():
978+ """Get this unit's public IP address"""
979+ return unit_get('public-address')
980+
981+
982+def unit_private_ip():
983+ """Get this unit's private IP address"""
984+ return unit_get('private-address')
985+
986+
987+@cached
988+def storage_get(attribute=None, storage_id=None):
989+ """Get storage attributes"""
990+ _args = ['storage-get', '--format=json']
991+ if storage_id:
992+ _args.extend(('-s', storage_id))
993+ if attribute:
994+ _args.append(attribute)
995+ try:
996+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
997+ except ValueError:
998+ return None
999+
1000+
1001+@cached
1002+def storage_list(storage_name=None):
1003+ """List the storage IDs for the unit"""
1004+ _args = ['storage-list', '--format=json']
1005+ if storage_name:
1006+ _args.append(storage_name)
1007+ try:
1008+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1009+ except ValueError:
1010+ return None
1011+ except OSError as e:
1012+ import errno
1013+ if e.errno == errno.ENOENT:
1014+ # storage-list does not exist
1015+ return []
1016+ raise
1017+
1018+
1019+class UnregisteredHookError(Exception):
1020+ """Raised when an undefined hook is called"""
1021+ pass
1022+
1023+
1024+class Hooks(object):
1025+ """A convenient handler for hook functions.
1026+
1027+ Example::
1028+
1029+ hooks = Hooks()
1030+
1031+ # register a hook, taking its name from the function name
1032+ @hooks.hook()
1033+ def install():
1034+ pass # your code here
1035+
1036+ # register a hook, providing a custom hook name
1037+ @hooks.hook("config-changed")
1038+ def config_changed():
1039+ pass # your code here
1040+
1041+ if __name__ == "__main__":
1042+ # execute a hook based on the name the program is called by
1043+ hooks.execute(sys.argv)
1044+ """
1045+
1046+ def __init__(self, config_save=None):
1047+ super(Hooks, self).__init__()
1048+ self._hooks = {}
1049+
1050+ # For unknown reasons, we allow the Hooks constructor to override
1051+ # config().implicit_save.
1052+ if config_save is not None:
1053+ config().implicit_save = config_save
1054+
1055+ def register(self, name, function):
1056+ """Register a hook"""
1057+ self._hooks[name] = function
1058+
1059+ def execute(self, args):
1060+ """Execute a registered hook based on args[0]"""
1061+ _run_atstart()
1062+ hook_name = os.path.basename(args[0])
1063+ if hook_name in self._hooks:
1064+ try:
1065+ self._hooks[hook_name]()
1066+ except SystemExit as x:
1067+ if x.code is None or x.code == 0:
1068+ _run_atexit()
1069+ raise
1070+ _run_atexit()
1071+ else:
1072+ raise UnregisteredHookError(hook_name)
1073+
1074+ def hook(self, *hook_names):
1075+ """Decorator, registering them as hooks"""
1076+ def wrapper(decorated):
1077+ for hook_name in hook_names:
1078+ self.register(hook_name, decorated)
1079+ else:
1080+ self.register(decorated.__name__, decorated)
1081+ if '_' in decorated.__name__:
1082+ self.register(
1083+ decorated.__name__.replace('_', '-'), decorated)
1084+ return decorated
1085+ return wrapper
1086+
1087+
1088+def charm_dir():
1089+ """Return the root directory of the current charm"""
1090+ return os.environ.get('CHARM_DIR')
1091+
1092+
1093+@cached
1094+def action_get(key=None):
1095+ """Gets the value of an action parameter, or all key/value param pairs"""
1096+ cmd = ['action-get']
1097+ if key is not None:
1098+ cmd.append(key)
1099+ cmd.append('--format=json')
1100+ action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1101+ return action_data
1102+
1103+
1104+def action_set(values):
1105+ """Sets the values to be returned after the action finishes"""
1106+ cmd = ['action-set']
1107+ for k, v in list(values.items()):
1108+ cmd.append('{}={}'.format(k, v))
1109+ subprocess.check_call(cmd)
1110+
1111+
1112+def action_fail(message):
1113+ """Sets the action status to failed and sets the error message.
1114+
1115+ The results set by action_set are preserved."""
1116+ subprocess.check_call(['action-fail', message])
1117+
1118+
1119+def action_name():
1120+ """Get the name of the currently executing action."""
1121+ return os.environ.get('JUJU_ACTION_NAME')
1122+
1123+
1124+def action_uuid():
1125+ """Get the UUID of the currently executing action."""
1126+ return os.environ.get('JUJU_ACTION_UUID')
1127+
1128+
1129+def action_tag():
1130+ """Get the tag for the currently executing action."""
1131+ return os.environ.get('JUJU_ACTION_TAG')
1132+
1133+
1134+def status_set(workload_state, message):
1135+ """Set the workload state with a message
1136+
1137+ Use status-set to set the workload state with a message which is visible
1138+ to the user via juju status. If the status-set command is not found then
1139+ assume this is juju < 1.23 and juju-log the message unstead.
1140+
1141+ workload_state -- valid juju workload state.
1142+ message -- status update message
1143+ """
1144+ valid_states = ['maintenance', 'blocked', 'waiting', 'active']
1145+ if workload_state not in valid_states:
1146+ raise ValueError(
1147+ '{!r} is not a valid workload state'.format(workload_state)
1148+ )
1149+ cmd = ['status-set', workload_state, message]
1150+ try:
1151+ ret = subprocess.call(cmd)
1152+ if ret == 0:
1153+ return
1154+ except OSError as e:
1155+ if e.errno != errno.ENOENT:
1156+ raise
1157+ log_message = 'status-set failed: {} {}'.format(workload_state,
1158+ message)
1159+ log(log_message, level='INFO')
1160+
1161+
1162+def status_get():
1163+ """Retrieve the previously set juju workload state and message
1164+
1165+ If the status-get command is not found then assume this is juju < 1.23 and
1166+ return 'unknown', ""
1167+
1168+ """
1169+ cmd = ['status-get', "--format=json", "--include-data"]
1170+ try:
1171+ raw_status = subprocess.check_output(cmd)
1172+ except OSError as e:
1173+ if e.errno == errno.ENOENT:
1174+ return ('unknown', "")
1175+ else:
1176+ raise
1177+ else:
1178+ status = json.loads(raw_status.decode("UTF-8"))
1179+ return (status["status"], status["message"])
1180+
1181+
1182+def translate_exc(from_exc, to_exc):
1183+ def inner_translate_exc1(f):
1184+ @wraps(f)
1185+ def inner_translate_exc2(*args, **kwargs):
1186+ try:
1187+ return f(*args, **kwargs)
1188+ except from_exc:
1189+ raise to_exc
1190+
1191+ return inner_translate_exc2
1192+
1193+ return inner_translate_exc1
1194+
1195+
1196+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1197+def is_leader():
1198+ """Does the current unit hold the juju leadership
1199+
1200+ Uses juju to determine whether the current unit is the leader of its peers
1201+ """
1202+ cmd = ['is-leader', '--format=json']
1203+ return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1204+
1205+
1206+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1207+def leader_get(attribute=None):
1208+ """Juju leader get value(s)"""
1209+ cmd = ['leader-get', '--format=json'] + [attribute or '-']
1210+ return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1211+
1212+
1213+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1214+def leader_set(settings=None, **kwargs):
1215+ """Juju leader set value(s)"""
1216+ # Don't log secrets.
1217+ # log("Juju leader-set '%s'" % (settings), level=DEBUG)
1218+ cmd = ['leader-set']
1219+ settings = settings or {}
1220+ settings.update(kwargs)
1221+ for k, v in settings.items():
1222+ if v is None:
1223+ cmd.append('{}='.format(k))
1224+ else:
1225+ cmd.append('{}={}'.format(k, v))
1226+ subprocess.check_call(cmd)
1227+
1228+
1229+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1230+def payload_register(ptype, klass, pid):
1231+ """ is used while a hook is running to let Juju know that a
1232+ payload has been started."""
1233+ cmd = ['payload-register']
1234+ for x in [ptype, klass, pid]:
1235+ cmd.append(x)
1236+ subprocess.check_call(cmd)
1237+
1238+
1239+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1240+def payload_unregister(klass, pid):
1241+ """ is used while a hook is running to let Juju know
1242+ that a payload has been manually stopped. The <class> and <id> provided
1243+ must match a payload that has been previously registered with juju using
1244+ payload-register."""
1245+ cmd = ['payload-unregister']
1246+ for x in [klass, pid]:
1247+ cmd.append(x)
1248+ subprocess.check_call(cmd)
1249+
1250+
1251+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1252+def payload_status_set(klass, pid, status):
1253+ """is used to update the current status of a registered payload.
1254+ The <class> and <id> provided must match a payload that has been previously
1255+ registered with juju using payload-register. The <status> must be one of the
1256+ follow: starting, started, stopping, stopped"""
1257+ cmd = ['payload-status-set']
1258+ for x in [klass, pid, status]:
1259+ cmd.append(x)
1260+ subprocess.check_call(cmd)
1261+
1262+
1263+@cached
1264+def juju_version():
1265+ """Full version string (eg. '1.23.3.1-trusty-amd64')"""
1266+ # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
1267+ jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
1268+ return subprocess.check_output([jujud, 'version'],
1269+ universal_newlines=True).strip()
1270+
1271+
1272+@cached
1273+def has_juju_version(minimum_version):
1274+ """Return True if the Juju version is at least the provided version"""
1275+ return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
1276+
1277+
1278+_atexit = []
1279+_atstart = []
1280+
1281+
1282+def atstart(callback, *args, **kwargs):
1283+ '''Schedule a callback to run before the main hook.
1284+
1285+ Callbacks are run in the order they were added.
1286+
1287+ This is useful for modules and classes to perform initialization
1288+ and inject behavior. In particular:
1289+
1290+ - Run common code before all of your hooks, such as logging
1291+ the hook name or interesting relation data.
1292+ - Defer object or module initialization that requires a hook
1293+ context until we know there actually is a hook context,
1294+ making testing easier.
1295+ - Rather than requiring charm authors to include boilerplate to
1296+ invoke your helper's behavior, have it run automatically if
1297+ your object is instantiated or module imported.
1298+
1299+ This is not at all useful after your hook framework as been launched.
1300+ '''
1301+ global _atstart
1302+ _atstart.append((callback, args, kwargs))
1303+
1304+
1305+def atexit(callback, *args, **kwargs):
1306+ '''Schedule a callback to run on successful hook completion.
1307+
1308+ Callbacks are run in the reverse order that they were added.'''
1309+ _atexit.append((callback, args, kwargs))
1310+
1311+
1312+def _run_atstart():
1313+ '''Hook frameworks must invoke this before running the main hook body.'''
1314+ global _atstart
1315+ for callback, args, kwargs in _atstart:
1316+ callback(*args, **kwargs)
1317+ del _atstart[:]
1318+
1319+
1320+def _run_atexit():
1321+ '''Hook frameworks must invoke this after the main hook body has
1322+ successfully completed. Do not invoke it if the hook fails.'''
1323+ global _atexit
1324+ for callback, args, kwargs in reversed(_atexit):
1325+ callback(*args, **kwargs)
1326+ del _atexit[:]
1327
1328=== added file 'hooks/charmhelpers/core/host.py'
1329--- hooks/charmhelpers/core/host.py 1970-01-01 00:00:00 +0000
1330+++ hooks/charmhelpers/core/host.py 2015-12-17 05:44:32 +0000
1331@@ -0,0 +1,641 @@
1332+# Copyright 2014-2015 Canonical Limited.
1333+#
1334+# This file is part of charm-helpers.
1335+#
1336+# charm-helpers is free software: you can redistribute it and/or modify
1337+# it under the terms of the GNU Lesser General Public License version 3 as
1338+# published by the Free Software Foundation.
1339+#
1340+# charm-helpers is distributed in the hope that it will be useful,
1341+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1342+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1343+# GNU Lesser General Public License for more details.
1344+#
1345+# You should have received a copy of the GNU Lesser General Public License
1346+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1347+
1348+"""Tools for working with the host system"""
1349+# Copyright 2012 Canonical Ltd.
1350+#
1351+# Authors:
1352+# Nick Moffitt <nick.moffitt@canonical.com>
1353+# Matthew Wedgwood <matthew.wedgwood@canonical.com>
1354+
1355+import os
1356+import re
1357+import pwd
1358+import glob
1359+import grp
1360+import random
1361+import string
1362+import subprocess
1363+import hashlib
1364+from contextlib import contextmanager
1365+from collections import OrderedDict
1366+
1367+import six
1368+
1369+from .hookenv import log
1370+from .fstab import Fstab
1371+
1372+
1373+def service_start(service_name):
1374+ """Start a system service"""
1375+ return service('start', service_name)
1376+
1377+
1378+def service_stop(service_name):
1379+ """Stop a system service"""
1380+ return service('stop', service_name)
1381+
1382+
1383+def service_restart(service_name):
1384+ """Restart a system service"""
1385+ return service('restart', service_name)
1386+
1387+
1388+def service_reload(service_name, restart_on_failure=False):
1389+ """Reload a system service, optionally falling back to restart if
1390+ reload fails"""
1391+ service_result = service('reload', service_name)
1392+ if not service_result and restart_on_failure:
1393+ service_result = service('restart', service_name)
1394+ return service_result
1395+
1396+
1397+def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"):
1398+ """Pause a system service.
1399+
1400+ Stop it, and prevent it from starting again at boot."""
1401+ stopped = True
1402+ if service_running(service_name):
1403+ stopped = service_stop(service_name)
1404+ upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
1405+ sysv_file = os.path.join(initd_dir, service_name)
1406+ if os.path.exists(upstart_file):
1407+ override_path = os.path.join(
1408+ init_dir, '{}.override'.format(service_name))
1409+ with open(override_path, 'w') as fh:
1410+ fh.write("manual\n")
1411+ elif os.path.exists(sysv_file):
1412+ subprocess.check_call(["update-rc.d", service_name, "disable"])
1413+ else:
1414+ # XXX: Support SystemD too
1415+ raise ValueError(
1416+ "Unable to detect {0} as either Upstart {1} or SysV {2}".format(
1417+ service_name, upstart_file, sysv_file))
1418+ return stopped
1419+
1420+
1421+def service_resume(service_name, init_dir="/etc/init",
1422+ initd_dir="/etc/init.d"):
1423+ """Resume a system service.
1424+
1425+ Reenable starting again at boot. Start the service"""
1426+ upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
1427+ sysv_file = os.path.join(initd_dir, service_name)
1428+ if os.path.exists(upstart_file):
1429+ override_path = os.path.join(
1430+ init_dir, '{}.override'.format(service_name))
1431+ if os.path.exists(override_path):
1432+ os.unlink(override_path)
1433+ elif os.path.exists(sysv_file):
1434+ subprocess.check_call(["update-rc.d", service_name, "enable"])
1435+ else:
1436+ # XXX: Support SystemD too
1437+ raise ValueError(
1438+ "Unable to detect {0} as either Upstart {1} or SysV {2}".format(
1439+ service_name, upstart_file, sysv_file))
1440+
1441+ started = service_running(service_name)
1442+ if not started:
1443+ started = service_start(service_name)
1444+ return started
1445+
1446+
1447+def service(action, service_name):
1448+ """Control a system service"""
1449+ cmd = ['service', service_name, action]
1450+ return subprocess.call(cmd) == 0
1451+
1452+
1453+def service_running(service):
1454+ """Determine whether a system service is running"""
1455+ try:
1456+ output = subprocess.check_output(
1457+ ['service', service, 'status'],
1458+ stderr=subprocess.STDOUT).decode('UTF-8')
1459+ except subprocess.CalledProcessError:
1460+ return False
1461+ else:
1462+ if ("start/running" in output or "is running" in output):
1463+ return True
1464+ else:
1465+ return False
1466+
1467+
1468+def service_available(service_name):
1469+ """Determine whether a system service is available"""
1470+ try:
1471+ subprocess.check_output(
1472+ ['service', service_name, 'status'],
1473+ stderr=subprocess.STDOUT).decode('UTF-8')
1474+ except subprocess.CalledProcessError as e:
1475+ return b'unrecognized service' not in e.output
1476+ else:
1477+ return True
1478+
1479+
1480+def adduser(username, password=None, shell='/bin/bash', system_user=False,
1481+ primary_group=None, secondary_groups=None):
1482+ """
1483+ Add a user to the system.
1484+
1485+ Will log but otherwise succeed if the user already exists.
1486+
1487+ :param str username: Username to create
1488+ :param str password: Password for user; if ``None``, create a system user
1489+ :param str shell: The default shell for the user
1490+ :param bool system_user: Whether to create a login or system user
1491+ :param str primary_group: Primary group for user; defaults to their username
1492+ :param list secondary_groups: Optional list of additional groups
1493+
1494+ :returns: The password database entry struct, as returned by `pwd.getpwnam`
1495+ """
1496+ try:
1497+ user_info = pwd.getpwnam(username)
1498+ log('user {0} already exists!'.format(username))
1499+ except KeyError:
1500+ log('creating user {0}'.format(username))
1501+ cmd = ['useradd']
1502+ if system_user or password is None:
1503+ cmd.append('--system')
1504+ else:
1505+ cmd.extend([
1506+ '--create-home',
1507+ '--shell', shell,
1508+ '--password', password,
1509+ ])
1510+ if not primary_group:
1511+ try:
1512+ grp.getgrnam(username)
1513+ primary_group = username # avoid "group exists" error
1514+ except KeyError:
1515+ pass
1516+ if primary_group:
1517+ cmd.extend(['-g', primary_group])
1518+ if secondary_groups:
1519+ cmd.extend(['-G', ','.join(secondary_groups)])
1520+ cmd.append(username)
1521+ subprocess.check_call(cmd)
1522+ user_info = pwd.getpwnam(username)
1523+ return user_info
1524+
1525+
1526+def user_exists(username):
1527+ """Check if a user exists"""
1528+ try:
1529+ pwd.getpwnam(username)
1530+ user_exists = True
1531+ except KeyError:
1532+ user_exists = False
1533+ return user_exists
1534+
1535+
1536+def add_group(group_name, system_group=False):
1537+ """Add a group to the system"""
1538+ try:
1539+ group_info = grp.getgrnam(group_name)
1540+ log('group {0} already exists!'.format(group_name))
1541+ except KeyError:
1542+ log('creating group {0}'.format(group_name))
1543+ cmd = ['addgroup']
1544+ if system_group:
1545+ cmd.append('--system')
1546+ else:
1547+ cmd.extend([
1548+ '--group',
1549+ ])
1550+ cmd.append(group_name)
1551+ subprocess.check_call(cmd)
1552+ group_info = grp.getgrnam(group_name)
1553+ return group_info
1554+
1555+
1556+def add_user_to_group(username, group):
1557+ """Add a user to a group"""
1558+ cmd = ['gpasswd', '-a', username, group]
1559+ log("Adding user {} to group {}".format(username, group))
1560+ subprocess.check_call(cmd)
1561+
1562+
1563+def rsync(from_path, to_path, flags='-r', options=None):
1564+ """Replicate the contents of a path"""
1565+ options = options or ['--delete', '--executability']
1566+ cmd = ['/usr/bin/rsync', flags]
1567+ cmd.extend(options)
1568+ cmd.append(from_path)
1569+ cmd.append(to_path)
1570+ log(" ".join(cmd))
1571+ return subprocess.check_output(cmd).decode('UTF-8').strip()
1572+
1573+
1574+def symlink(source, destination):
1575+ """Create a symbolic link"""
1576+ log("Symlinking {} as {}".format(source, destination))
1577+ cmd = [
1578+ 'ln',
1579+ '-sf',
1580+ source,
1581+ destination,
1582+ ]
1583+ subprocess.check_call(cmd)
1584+
1585+
1586+def mkdir(path, owner='root', group='root', perms=0o555, force=False):
1587+ """Create a directory"""
1588+ log("Making dir {} {}:{} {:o}".format(path, owner, group,
1589+ perms))
1590+ uid = pwd.getpwnam(owner).pw_uid
1591+ gid = grp.getgrnam(group).gr_gid
1592+ realpath = os.path.abspath(path)
1593+ path_exists = os.path.exists(realpath)
1594+ if path_exists and force:
1595+ if not os.path.isdir(realpath):
1596+ log("Removing non-directory file {} prior to mkdir()".format(path))
1597+ os.unlink(realpath)
1598+ os.makedirs(realpath, perms)
1599+ elif not path_exists:
1600+ os.makedirs(realpath, perms)
1601+ os.chown(realpath, uid, gid)
1602+ os.chmod(realpath, perms)
1603+
1604+
1605+def write_file(path, content, owner='root', group='root', perms=0o444):
1606+ """Create or overwrite a file with the contents of a byte string."""
1607+ log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
1608+ uid = pwd.getpwnam(owner).pw_uid
1609+ gid = grp.getgrnam(group).gr_gid
1610+ with open(path, 'wb') as target:
1611+ os.fchown(target.fileno(), uid, gid)
1612+ os.fchmod(target.fileno(), perms)
1613+ target.write(content)
1614+
1615+
1616+def fstab_remove(mp):
1617+ """Remove the given mountpoint entry from /etc/fstab
1618+ """
1619+ return Fstab.remove_by_mountpoint(mp)
1620+
1621+
1622+def fstab_add(dev, mp, fs, options=None):
1623+ """Adds the given device entry to the /etc/fstab file
1624+ """
1625+ return Fstab.add(dev, mp, fs, options=options)
1626+
1627+
1628+def mount(device, mountpoint, options=None, persist=False, filesystem="ext3"):
1629+ """Mount a filesystem at a particular mountpoint"""
1630+ cmd_args = ['mount']
1631+ if options is not None:
1632+ cmd_args.extend(['-o', options])
1633+ cmd_args.extend([device, mountpoint])
1634+ try:
1635+ subprocess.check_output(cmd_args)
1636+ except subprocess.CalledProcessError as e:
1637+ log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
1638+ return False
1639+
1640+ if persist:
1641+ return fstab_add(device, mountpoint, filesystem, options=options)
1642+ return True
1643+
1644+
1645+def umount(mountpoint, persist=False):
1646+ """Unmount a filesystem"""
1647+ cmd_args = ['umount', mountpoint]
1648+ try:
1649+ subprocess.check_output(cmd_args)
1650+ except subprocess.CalledProcessError as e:
1651+ log('Error unmounting {}\n{}'.format(mountpoint, e.output))
1652+ return False
1653+
1654+ if persist:
1655+ return fstab_remove(mountpoint)
1656+ return True
1657+
1658+
1659+def mounts():
1660+ """Get a list of all mounted volumes as [[mountpoint,device],[...]]"""
1661+ with open('/proc/mounts') as f:
1662+ # [['/mount/point','/dev/path'],[...]]
1663+ system_mounts = [m[1::-1] for m in [l.strip().split()
1664+ for l in f.readlines()]]
1665+ return system_mounts
1666+
1667+
1668+def fstab_mount(mountpoint):
1669+ """Mount filesystem using fstab"""
1670+ cmd_args = ['mount', mountpoint]
1671+ try:
1672+ subprocess.check_output(cmd_args)
1673+ except subprocess.CalledProcessError as e:
1674+ log('Error unmounting {}\n{}'.format(mountpoint, e.output))
1675+ return False
1676+ return True
1677+
1678+
1679+def file_hash(path, hash_type='md5'):
1680+ """
1681+ Generate a hash checksum of the contents of 'path' or None if not found.
1682+
1683+ :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
1684+ such as md5, sha1, sha256, sha512, etc.
1685+ """
1686+ if os.path.exists(path):
1687+ h = getattr(hashlib, hash_type)()
1688+ with open(path, 'rb') as source:
1689+ h.update(source.read())
1690+ return h.hexdigest()
1691+ else:
1692+ return None
1693+
1694+
1695+def path_hash(path):
1696+ """
1697+ Generate a hash checksum of all files matching 'path'. Standard wildcards
1698+ like '*' and '?' are supported, see documentation for the 'glob' module for
1699+ more information.
1700+
1701+ :return: dict: A { filename: hash } dictionary for all matched files.
1702+ Empty if none found.
1703+ """
1704+ return {
1705+ filename: file_hash(filename)
1706+ for filename in glob.iglob(path)
1707+ }
1708+
1709+
1710+def check_hash(path, checksum, hash_type='md5'):
1711+ """
1712+ Validate a file using a cryptographic checksum.
1713+
1714+ :param str checksum: Value of the checksum used to validate the file.
1715+ :param str hash_type: Hash algorithm used to generate `checksum`.
1716+ Can be any hash alrgorithm supported by :mod:`hashlib`,
1717+ such as md5, sha1, sha256, sha512, etc.
1718+ :raises ChecksumError: If the file fails the checksum
1719+
1720+ """
1721+ actual_checksum = file_hash(path, hash_type)
1722+ if checksum != actual_checksum:
1723+ raise ChecksumError("'%s' != '%s'" % (checksum, actual_checksum))
1724+
1725+
1726+class ChecksumError(ValueError):
1727+ pass
1728+
1729+
1730+def restart_on_change(restart_map, stopstart=False):
1731+ """Restart services based on configuration files changing
1732+
1733+ This function is used a decorator, for example::
1734+
1735+ @restart_on_change({
1736+ '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
1737+ '/etc/apache/sites-enabled/*': [ 'apache2' ]
1738+ })
1739+ def config_changed():
1740+ pass # your code here
1741+
1742+ In this example, the cinder-api and cinder-volume services
1743+ would be restarted if /etc/ceph/ceph.conf is changed by the
1744+ ceph_client_changed function. The apache2 service would be
1745+ restarted if any file matching the pattern got changed, created
1746+ or removed. Standard wildcards are supported, see documentation
1747+ for the 'glob' module for more information.
1748+ """
1749+ def wrap(f):
1750+ def wrapped_f(*args, **kwargs):
1751+ checksums = {path: path_hash(path) for path in restart_map}
1752+ f(*args, **kwargs)
1753+ restarts = []
1754+ for path in restart_map:
1755+ if path_hash(path) != checksums[path]:
1756+ restarts += restart_map[path]
1757+ services_list = list(OrderedDict.fromkeys(restarts))
1758+ if not stopstart:
1759+ for service_name in services_list:
1760+ service('restart', service_name)
1761+ else:
1762+ for action in ['stop', 'start']:
1763+ for service_name in services_list:
1764+ service(action, service_name)
1765+ return wrapped_f
1766+ return wrap
1767+
1768+
1769+def lsb_release():
1770+ """Return /etc/lsb-release in a dict"""
1771+ d = {}
1772+ with open('/etc/lsb-release', 'r') as lsb:
1773+ for l in lsb:
1774+ k, v = l.split('=')
1775+ d[k.strip()] = v.strip()
1776+ return d
1777+
1778+
1779+def pwgen(length=None):
1780+ """Generate a random pasword."""
1781+ if length is None:
1782+ # A random length is ok to use a weak PRNG
1783+ length = random.choice(range(35, 45))
1784+ alphanumeric_chars = [
1785+ l for l in (string.ascii_letters + string.digits)
1786+ if l not in 'l0QD1vAEIOUaeiou']
1787+ # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
1788+ # actual password
1789+ random_generator = random.SystemRandom()
1790+ random_chars = [
1791+ random_generator.choice(alphanumeric_chars) for _ in range(length)]
1792+ return(''.join(random_chars))
1793+
1794+
1795+def is_phy_iface(interface):
1796+ """Returns True if interface is not virtual, otherwise False."""
1797+ if interface:
1798+ sys_net = '/sys/class/net'
1799+ if os.path.isdir(sys_net):
1800+ for iface in glob.glob(os.path.join(sys_net, '*')):
1801+ if '/virtual/' in os.path.realpath(iface):
1802+ continue
1803+
1804+ if interface == os.path.basename(iface):
1805+ return True
1806+
1807+ return False
1808+
1809+
1810+def get_bond_master(interface):
1811+ """Returns bond master if interface is bond slave otherwise None.
1812+
1813+ NOTE: the provided interface is expected to be physical
1814+ """
1815+ if interface:
1816+ iface_path = '/sys/class/net/%s' % (interface)
1817+ if os.path.exists(iface_path):
1818+ if '/virtual/' in os.path.realpath(iface_path):
1819+ return None
1820+
1821+ master = os.path.join(iface_path, 'master')
1822+ if os.path.exists(master):
1823+ master = os.path.realpath(master)
1824+ # make sure it is a bond master
1825+ if os.path.exists(os.path.join(master, 'bonding')):
1826+ return os.path.basename(master)
1827+
1828+ return None
1829+
1830+
1831+def list_nics(nic_type=None):
1832+ '''Return a list of nics of given type(s)'''
1833+ if isinstance(nic_type, six.string_types):
1834+ int_types = [nic_type]
1835+ else:
1836+ int_types = nic_type
1837+
1838+ interfaces = []
1839+ if nic_type:
1840+ for int_type in int_types:
1841+ cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
1842+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
1843+ ip_output = ip_output.split('\n')
1844+ ip_output = (line for line in ip_output if line)
1845+ for line in ip_output:
1846+ if line.split()[1].startswith(int_type):
1847+ matched = re.search('.*: (' + int_type +
1848+ r'[0-9]+\.[0-9]+)@.*', line)
1849+ if matched:
1850+ iface = matched.groups()[0]
1851+ else:
1852+ iface = line.split()[1].replace(":", "")
1853+
1854+ if iface not in interfaces:
1855+ interfaces.append(iface)
1856+ else:
1857+ cmd = ['ip', 'a']
1858+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1859+ ip_output = (line.strip() for line in ip_output if line)
1860+
1861+ key = re.compile('^[0-9]+:\s+(.+):')
1862+ for line in ip_output:
1863+ matched = re.search(key, line)
1864+ if matched:
1865+ iface = matched.group(1)
1866+ iface = iface.partition("@")[0]
1867+ if iface not in interfaces:
1868+ interfaces.append(iface)
1869+
1870+ return interfaces
1871+
1872+
1873+def set_nic_mtu(nic, mtu):
1874+ '''Set MTU on a network interface'''
1875+ cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]
1876+ subprocess.check_call(cmd)
1877+
1878+
1879+def get_nic_mtu(nic):
1880+ cmd = ['ip', 'addr', 'show', nic]
1881+ ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1882+ mtu = ""
1883+ for line in ip_output:
1884+ words = line.split()
1885+ if 'mtu' in words:
1886+ mtu = words[words.index("mtu") + 1]
1887+ return mtu
1888+
1889+
1890+def get_nic_hwaddr(nic):
1891+ cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
1892+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
1893+ hwaddr = ""
1894+ words = ip_output.split()
1895+ if 'link/ether' in words:
1896+ hwaddr = words[words.index('link/ether') + 1]
1897+ return hwaddr
1898+
1899+
1900+def cmp_pkgrevno(package, revno, pkgcache=None):
1901+ '''Compare supplied revno with the revno of the installed package
1902+
1903+ * 1 => Installed revno is greater than supplied arg
1904+ * 0 => Installed revno is the same as supplied arg
1905+ * -1 => Installed revno is less than supplied arg
1906+
1907+ This function imports apt_cache function from charmhelpers.fetch if
1908+ the pkgcache argument is None. Be sure to add charmhelpers.fetch if
1909+ you call this function, or pass an apt_pkg.Cache() instance.
1910+ '''
1911+ import apt_pkg
1912+ if not pkgcache:
1913+ from charmhelpers.fetch import apt_cache
1914+ pkgcache = apt_cache()
1915+ pkg = pkgcache[package]
1916+ return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
1917+
1918+
1919+@contextmanager
1920+def chdir(d):
1921+ cur = os.getcwd()
1922+ try:
1923+ yield os.chdir(d)
1924+ finally:
1925+ os.chdir(cur)
1926+
1927+
1928+def chownr(path, owner, group, follow_links=True, chowntopdir=False):
1929+ """
1930+ Recursively change user and group ownership of files and directories
1931+ in given path. Doesn't chown path itself by default, only its children.
1932+
1933+ :param bool follow_links: Also Chown links if True
1934+ :param bool chowntopdir: Also chown path itself if True
1935+ """
1936+ uid = pwd.getpwnam(owner).pw_uid
1937+ gid = grp.getgrnam(group).gr_gid
1938+ if follow_links:
1939+ chown = os.chown
1940+ else:
1941+ chown = os.lchown
1942+
1943+ if chowntopdir:
1944+ broken_symlink = os.path.lexists(path) and not os.path.exists(path)
1945+ if not broken_symlink:
1946+ chown(path, uid, gid)
1947+ for root, dirs, files in os.walk(path):
1948+ for name in dirs + files:
1949+ full = os.path.join(root, name)
1950+ broken_symlink = os.path.lexists(full) and not os.path.exists(full)
1951+ if not broken_symlink:
1952+ chown(full, uid, gid)
1953+
1954+
1955+def lchownr(path, owner, group):
1956+ chownr(path, owner, group, follow_links=False)
1957+
1958+
1959+def get_total_ram():
1960+ '''The total amount of system RAM in bytes.
1961+
1962+ This is what is reported by the OS, and may be overcommitted when
1963+ there are multiple containers hosted on the same machine.
1964+ '''
1965+ with open('/proc/meminfo', 'r') as f:
1966+ for line in f.readlines():
1967+ if line:
1968+ key, value, unit = line.split()
1969+ if key == 'MemTotal:':
1970+ assert unit == 'kB', 'Unknown unit'
1971+ return int(value) * 1024 # Classic, not KiB.
1972+ raise NotImplementedError()
1973
1974=== added file 'hooks/charmhelpers/core/hugepage.py'
1975--- hooks/charmhelpers/core/hugepage.py 1970-01-01 00:00:00 +0000
1976+++ hooks/charmhelpers/core/hugepage.py 2015-12-17 05:44:32 +0000
1977@@ -0,0 +1,71 @@
1978+# -*- coding: utf-8 -*-
1979+
1980+# Copyright 2014-2015 Canonical Limited.
1981+#
1982+# This file is part of charm-helpers.
1983+#
1984+# charm-helpers is free software: you can redistribute it and/or modify
1985+# it under the terms of the GNU Lesser General Public License version 3 as
1986+# published by the Free Software Foundation.
1987+#
1988+# charm-helpers is distributed in the hope that it will be useful,
1989+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1990+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1991+# GNU Lesser General Public License for more details.
1992+#
1993+# You should have received a copy of the GNU Lesser General Public License
1994+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1995+
1996+import yaml
1997+from charmhelpers.core import fstab
1998+from charmhelpers.core import sysctl
1999+from charmhelpers.core.host import (
2000+ add_group,
2001+ add_user_to_group,
2002+ fstab_mount,
2003+ mkdir,
2004+)
2005+from charmhelpers.core.strutils import bytes_from_string
2006+from subprocess import check_output
2007+
2008+
2009+def hugepage_support(user, group='hugetlb', nr_hugepages=256,
2010+ max_map_count=65536, mnt_point='/run/hugepages/kvm',
2011+ pagesize='2MB', mount=True, set_shmmax=False):
2012+ """Enable hugepages on system.
2013+
2014+ Args:
2015+ user (str) -- Username to allow access to hugepages to
2016+ group (str) -- Group name to own hugepages
2017+ nr_hugepages (int) -- Number of pages to reserve
2018+ max_map_count (int) -- Number of Virtual Memory Areas a process can own
2019+ mnt_point (str) -- Directory to mount hugepages on
2020+ pagesize (str) -- Size of hugepages
2021+ mount (bool) -- Whether to Mount hugepages
2022+ """
2023+ group_info = add_group(group)
2024+ gid = group_info.gr_gid
2025+ add_user_to_group(user, group)
2026+ if max_map_count < 2 * nr_hugepages:
2027+ max_map_count = 2 * nr_hugepages
2028+ sysctl_settings = {
2029+ 'vm.nr_hugepages': nr_hugepages,
2030+ 'vm.max_map_count': max_map_count,
2031+ 'vm.hugetlb_shm_group': gid,
2032+ }
2033+ if set_shmmax:
2034+ shmmax_current = int(check_output(['sysctl', '-n', 'kernel.shmmax']))
2035+ shmmax_minsize = bytes_from_string(pagesize) * nr_hugepages
2036+ if shmmax_minsize > shmmax_current:
2037+ sysctl_settings['kernel.shmmax'] = shmmax_minsize
2038+ sysctl.create(yaml.dump(sysctl_settings), '/etc/sysctl.d/10-hugepage.conf')
2039+ mkdir(mnt_point, owner='root', group='root', perms=0o755, force=False)
2040+ lfstab = fstab.Fstab()
2041+ fstab_entry = lfstab.get_entry_by_attr('mountpoint', mnt_point)
2042+ if fstab_entry:
2043+ lfstab.remove_entry(fstab_entry)
2044+ entry = lfstab.Entry('nodev', mnt_point, 'hugetlbfs',
2045+ 'mode=1770,gid={},pagesize={}'.format(gid, pagesize), 0, 0)
2046+ lfstab.add_entry(entry)
2047+ if mount:
2048+ fstab_mount(mnt_point)
2049
2050=== added file 'hooks/charmhelpers/core/kernel.py'
2051--- hooks/charmhelpers/core/kernel.py 1970-01-01 00:00:00 +0000
2052+++ hooks/charmhelpers/core/kernel.py 2015-12-17 05:44:32 +0000
2053@@ -0,0 +1,68 @@
2054+#!/usr/bin/env python
2055+# -*- coding: utf-8 -*-
2056+
2057+# Copyright 2014-2015 Canonical Limited.
2058+#
2059+# This file is part of charm-helpers.
2060+#
2061+# charm-helpers is free software: you can redistribute it and/or modify
2062+# it under the terms of the GNU Lesser General Public License version 3 as
2063+# published by the Free Software Foundation.
2064+#
2065+# charm-helpers is distributed in the hope that it will be useful,
2066+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2067+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2068+# GNU Lesser General Public License for more details.
2069+#
2070+# You should have received a copy of the GNU Lesser General Public License
2071+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2072+
2073+__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
2074+
2075+from charmhelpers.core.hookenv import (
2076+ log,
2077+ INFO
2078+)
2079+
2080+from subprocess import check_call, check_output
2081+import re
2082+
2083+
2084+def modprobe(module, persist=True):
2085+ """Load a kernel module and configure for auto-load on reboot."""
2086+ cmd = ['modprobe', module]
2087+
2088+ log('Loading kernel module %s' % module, level=INFO)
2089+
2090+ check_call(cmd)
2091+ if persist:
2092+ with open('/etc/modules', 'r+') as modules:
2093+ if module not in modules.read():
2094+ modules.write(module)
2095+
2096+
2097+def rmmod(module, force=False):
2098+ """Remove a module from the linux kernel"""
2099+ cmd = ['rmmod']
2100+ if force:
2101+ cmd.append('-f')
2102+ cmd.append(module)
2103+ log('Removing kernel module %s' % module, level=INFO)
2104+ return check_call(cmd)
2105+
2106+
2107+def lsmod():
2108+ """Shows what kernel modules are currently loaded"""
2109+ return check_output(['lsmod'],
2110+ universal_newlines=True)
2111+
2112+
2113+def is_module_loaded(module):
2114+ """Checks if a kernel module is already loaded"""
2115+ matches = re.findall('^%s[ ]+' % module, lsmod(), re.M)
2116+ return len(matches) > 0
2117+
2118+
2119+def update_initramfs(version='all'):
2120+ """Updates an initramfs image"""
2121+ return check_call(["update-initramfs", "-k", version, "-u"])
2122
2123=== added directory 'hooks/charmhelpers/core/services'
2124=== added file 'hooks/charmhelpers/core/services/__init__.py'
2125--- hooks/charmhelpers/core/services/__init__.py 1970-01-01 00:00:00 +0000
2126+++ hooks/charmhelpers/core/services/__init__.py 2015-12-17 05:44:32 +0000
2127@@ -0,0 +1,18 @@
2128+# Copyright 2014-2015 Canonical Limited.
2129+#
2130+# This file is part of charm-helpers.
2131+#
2132+# charm-helpers is free software: you can redistribute it and/or modify
2133+# it under the terms of the GNU Lesser General Public License version 3 as
2134+# published by the Free Software Foundation.
2135+#
2136+# charm-helpers is distributed in the hope that it will be useful,
2137+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2138+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2139+# GNU Lesser General Public License for more details.
2140+#
2141+# You should have received a copy of the GNU Lesser General Public License
2142+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2143+
2144+from .base import * # NOQA
2145+from .helpers import * # NOQA
2146
2147=== added file 'hooks/charmhelpers/core/services/base.py'
2148--- hooks/charmhelpers/core/services/base.py 1970-01-01 00:00:00 +0000
2149+++ hooks/charmhelpers/core/services/base.py 2015-12-17 05:44:32 +0000
2150@@ -0,0 +1,353 @@
2151+# Copyright 2014-2015 Canonical Limited.
2152+#
2153+# This file is part of charm-helpers.
2154+#
2155+# charm-helpers is free software: you can redistribute it and/or modify
2156+# it under the terms of the GNU Lesser General Public License version 3 as
2157+# published by the Free Software Foundation.
2158+#
2159+# charm-helpers is distributed in the hope that it will be useful,
2160+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2161+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2162+# GNU Lesser General Public License for more details.
2163+#
2164+# You should have received a copy of the GNU Lesser General Public License
2165+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2166+
2167+import os
2168+import json
2169+from inspect import getargspec
2170+from collections import Iterable, OrderedDict
2171+
2172+from charmhelpers.core import host
2173+from charmhelpers.core import hookenv
2174+
2175+
2176+__all__ = ['ServiceManager', 'ManagerCallback',
2177+ 'PortManagerCallback', 'open_ports', 'close_ports', 'manage_ports',
2178+ 'service_restart', 'service_stop']
2179+
2180+
2181+class ServiceManager(object):
2182+ def __init__(self, services=None):
2183+ """
2184+ Register a list of services, given their definitions.
2185+
2186+ Service definitions are dicts in the following formats (all keys except
2187+ 'service' are optional)::
2188+
2189+ {
2190+ "service": <service name>,
2191+ "required_data": <list of required data contexts>,
2192+ "provided_data": <list of provided data contexts>,
2193+ "data_ready": <one or more callbacks>,
2194+ "data_lost": <one or more callbacks>,
2195+ "start": <one or more callbacks>,
2196+ "stop": <one or more callbacks>,
2197+ "ports": <list of ports to manage>,
2198+ }
2199+
2200+ The 'required_data' list should contain dicts of required data (or
2201+ dependency managers that act like dicts and know how to collect the data).
2202+ Only when all items in the 'required_data' list are populated are the list
2203+ of 'data_ready' and 'start' callbacks executed. See `is_ready()` for more
2204+ information.
2205+
2206+ The 'provided_data' list should contain relation data providers, most likely
2207+ a subclass of :class:`charmhelpers.core.services.helpers.RelationContext`,
2208+ that will indicate a set of data to set on a given relation.
2209+
2210+ The 'data_ready' value should be either a single callback, or a list of
2211+ callbacks, to be called when all items in 'required_data' pass `is_ready()`.
2212+ Each callback will be called with the service name as the only parameter.
2213+ After all of the 'data_ready' callbacks are called, the 'start' callbacks
2214+ are fired.
2215+
2216+ The 'data_lost' value should be either a single callback, or a list of
2217+ callbacks, to be called when a 'required_data' item no longer passes
2218+ `is_ready()`. Each callback will be called with the service name as the
2219+ only parameter. After all of the 'data_lost' callbacks are called,
2220+ the 'stop' callbacks are fired.
2221+
2222+ The 'start' value should be either a single callback, or a list of
2223+ callbacks, to be called when starting the service, after the 'data_ready'
2224+ callbacks are complete. Each callback will be called with the service
2225+ name as the only parameter. This defaults to
2226+ `[host.service_start, services.open_ports]`.
2227+
2228+ The 'stop' value should be either a single callback, or a list of
2229+ callbacks, to be called when stopping the service. If the service is
2230+ being stopped because it no longer has all of its 'required_data', this
2231+ will be called after all of the 'data_lost' callbacks are complete.
2232+ Each callback will be called with the service name as the only parameter.
2233+ This defaults to `[services.close_ports, host.service_stop]`.
2234+
2235+ The 'ports' value should be a list of ports to manage. The default
2236+ 'start' handler will open the ports after the service is started,
2237+ and the default 'stop' handler will close the ports prior to stopping
2238+ the service.
2239+
2240+
2241+ Examples:
2242+
2243+ The following registers an Upstart service called bingod that depends on
2244+ a mongodb relation and which runs a custom `db_migrate` function prior to
2245+ restarting the service, and a Runit service called spadesd::
2246+
2247+ manager = services.ServiceManager([
2248+ {
2249+ 'service': 'bingod',
2250+ 'ports': [80, 443],
2251+ 'required_data': [MongoRelation(), config(), {'my': 'data'}],
2252+ 'data_ready': [
2253+ services.template(source='bingod.conf'),
2254+ services.template(source='bingod.ini',
2255+ target='/etc/bingod.ini',
2256+ owner='bingo', perms=0400),
2257+ ],
2258+ },
2259+ {
2260+ 'service': 'spadesd',
2261+ 'data_ready': services.template(source='spadesd_run.j2',
2262+ target='/etc/sv/spadesd/run',
2263+ perms=0555),
2264+ 'start': runit_start,
2265+ 'stop': runit_stop,
2266+ },
2267+ ])
2268+ manager.manage()
2269+ """
2270+ self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
2271+ self._ready = None
2272+ self.services = OrderedDict()
2273+ for service in services or []:
2274+ service_name = service['service']
2275+ self.services[service_name] = service
2276+
2277+ def manage(self):
2278+ """
2279+ Handle the current hook by doing The Right Thing with the registered services.
2280+ """
2281+ hookenv._run_atstart()
2282+ try:
2283+ hook_name = hookenv.hook_name()
2284+ if hook_name == 'stop':
2285+ self.stop_services()
2286+ else:
2287+ self.reconfigure_services()
2288+ self.provide_data()
2289+ except SystemExit as x:
2290+ if x.code is None or x.code == 0:
2291+ hookenv._run_atexit()
2292+ hookenv._run_atexit()
2293+
2294+ def provide_data(self):
2295+ """
2296+ Set the relation data for each provider in the ``provided_data`` list.
2297+
2298+ A provider must have a `name` attribute, which indicates which relation
2299+ to set data on, and a `provide_data()` method, which returns a dict of
2300+ data to set.
2301+
2302+ The `provide_data()` method can optionally accept two parameters:
2303+
2304+ * ``remote_service`` The name of the remote service that the data will
2305+ be provided to. The `provide_data()` method will be called once
2306+ for each connected service (not unit). This allows the method to
2307+ tailor its data to the given service.
2308+ * ``service_ready`` Whether or not the service definition had all of
2309+ its requirements met, and thus the ``data_ready`` callbacks run.
2310+
2311+ Note that the ``provided_data`` methods are now called **after** the
2312+ ``data_ready`` callbacks are run. This gives the ``data_ready`` callbacks
2313+ a chance to generate any data necessary for the providing to the remote
2314+ services.
2315+ """
2316+ for service_name, service in self.services.items():
2317+ service_ready = self.is_ready(service_name)
2318+ for provider in service.get('provided_data', []):
2319+ for relid in hookenv.relation_ids(provider.name):
2320+ units = hookenv.related_units(relid)
2321+ if not units:
2322+ continue
2323+ remote_service = units[0].split('/')[0]
2324+ argspec = getargspec(provider.provide_data)
2325+ if len(argspec.args) > 1:
2326+ data = provider.provide_data(remote_service, service_ready)
2327+ else:
2328+ data = provider.provide_data()
2329+ if data:
2330+ hookenv.relation_set(relid, data)
2331+
2332+ def reconfigure_services(self, *service_names):
2333+ """
2334+ Update all files for one or more registered services, and,
2335+ if ready, optionally restart them.
2336+
2337+ If no service names are given, reconfigures all registered services.
2338+ """
2339+ for service_name in service_names or self.services.keys():
2340+ if self.is_ready(service_name):
2341+ self.fire_event('data_ready', service_name)
2342+ self.fire_event('start', service_name, default=[
2343+ service_restart,
2344+ manage_ports])
2345+ self.save_ready(service_name)
2346+ else:
2347+ if self.was_ready(service_name):
2348+ self.fire_event('data_lost', service_name)
2349+ self.fire_event('stop', service_name, default=[
2350+ manage_ports,
2351+ service_stop])
2352+ self.save_lost(service_name)
2353+
2354+ def stop_services(self, *service_names):
2355+ """
2356+ Stop one or more registered services, by name.
2357+
2358+ If no service names are given, stops all registered services.
2359+ """
2360+ for service_name in service_names or self.services.keys():
2361+ self.fire_event('stop', service_name, default=[
2362+ manage_ports,
2363+ service_stop])
2364+
2365+ def get_service(self, service_name):
2366+ """
2367+ Given the name of a registered service, return its service definition.
2368+ """
2369+ service = self.services.get(service_name)
2370+ if not service:
2371+ raise KeyError('Service not registered: %s' % service_name)
2372+ return service
2373+
2374+ def fire_event(self, event_name, service_name, default=None):
2375+ """
2376+ Fire a data_ready, data_lost, start, or stop event on a given service.
2377+ """
2378+ service = self.get_service(service_name)
2379+ callbacks = service.get(event_name, default)
2380+ if not callbacks:
2381+ return
2382+ if not isinstance(callbacks, Iterable):
2383+ callbacks = [callbacks]
2384+ for callback in callbacks:
2385+ if isinstance(callback, ManagerCallback):
2386+ callback(self, service_name, event_name)
2387+ else:
2388+ callback(service_name)
2389+
2390+ def is_ready(self, service_name):
2391+ """
2392+ Determine if a registered service is ready, by checking its 'required_data'.
2393+
2394+ A 'required_data' item can be any mapping type, and is considered ready
2395+ if `bool(item)` evaluates as True.
2396+ """
2397+ service = self.get_service(service_name)
2398+ reqs = service.get('required_data', [])
2399+ return all(bool(req) for req in reqs)
2400+
2401+ def _load_ready_file(self):
2402+ if self._ready is not None:
2403+ return
2404+ if os.path.exists(self._ready_file):
2405+ with open(self._ready_file) as fp:
2406+ self._ready = set(json.load(fp))
2407+ else:
2408+ self._ready = set()
2409+
2410+ def _save_ready_file(self):
2411+ if self._ready is None:
2412+ return
2413+ with open(self._ready_file, 'w') as fp:
2414+ json.dump(list(self._ready), fp)
2415+
2416+ def save_ready(self, service_name):
2417+ """
2418+ Save an indicator that the given service is now data_ready.
2419+ """
2420+ self._load_ready_file()
2421+ self._ready.add(service_name)
2422+ self._save_ready_file()
2423+
2424+ def save_lost(self, service_name):
2425+ """
2426+ Save an indicator that the given service is no longer data_ready.
2427+ """
2428+ self._load_ready_file()
2429+ self._ready.discard(service_name)
2430+ self._save_ready_file()
2431+
2432+ def was_ready(self, service_name):
2433+ """
2434+ Determine if the given service was previously data_ready.
2435+ """
2436+ self._load_ready_file()
2437+ return service_name in self._ready
2438+
2439+
2440+class ManagerCallback(object):
2441+ """
2442+ Special case of a callback that takes the `ServiceManager` instance
2443+ in addition to the service name.
2444+
2445+ Subclasses should implement `__call__` which should accept three parameters:
2446+
2447+ * `manager` The `ServiceManager` instance
2448+ * `service_name` The name of the service it's being triggered for
2449+ * `event_name` The name of the event that this callback is handling
2450+ """
2451+ def __call__(self, manager, service_name, event_name):
2452+ raise NotImplementedError()
2453+
2454+
2455+class PortManagerCallback(ManagerCallback):
2456+ """
2457+ Callback class that will open or close ports, for use as either
2458+ a start or stop action.
2459+ """
2460+ def __call__(self, manager, service_name, event_name):
2461+ service = manager.get_service(service_name)
2462+ new_ports = service.get('ports', [])
2463+ port_file = os.path.join(hookenv.charm_dir(), '.{}.ports'.format(service_name))
2464+ if os.path.exists(port_file):
2465+ with open(port_file) as fp:
2466+ old_ports = fp.read().split(',')
2467+ for old_port in old_ports:
2468+ if bool(old_port):
2469+ old_port = int(old_port)
2470+ if old_port not in new_ports:
2471+ hookenv.close_port(old_port)
2472+ with open(port_file, 'w') as fp:
2473+ fp.write(','.join(str(port) for port in new_ports))
2474+ for port in new_ports:
2475+ if event_name == 'start':
2476+ hookenv.open_port(port)
2477+ elif event_name == 'stop':
2478+ hookenv.close_port(port)
2479+
2480+
2481+def service_stop(service_name):
2482+ """
2483+ Wrapper around host.service_stop to prevent spurious "unknown service"
2484+ messages in the logs.
2485+ """
2486+ if host.service_running(service_name):
2487+ host.service_stop(service_name)
2488+
2489+
2490+def service_restart(service_name):
2491+ """
2492+ Wrapper around host.service_restart to prevent spurious "unknown service"
2493+ messages in the logs.
2494+ """
2495+ if host.service_available(service_name):
2496+ if host.service_running(service_name):
2497+ host.service_restart(service_name)
2498+ else:
2499+ host.service_start(service_name)
2500+
2501+
2502+# Convenience aliases
2503+open_ports = close_ports = manage_ports = PortManagerCallback()
2504
2505=== added file 'hooks/charmhelpers/core/services/helpers.py'
2506--- hooks/charmhelpers/core/services/helpers.py 1970-01-01 00:00:00 +0000
2507+++ hooks/charmhelpers/core/services/helpers.py 2015-12-17 05:44:32 +0000
2508@@ -0,0 +1,292 @@
2509+# Copyright 2014-2015 Canonical Limited.
2510+#
2511+# This file is part of charm-helpers.
2512+#
2513+# charm-helpers is free software: you can redistribute it and/or modify
2514+# it under the terms of the GNU Lesser General Public License version 3 as
2515+# published by the Free Software Foundation.
2516+#
2517+# charm-helpers is distributed in the hope that it will be useful,
2518+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2519+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2520+# GNU Lesser General Public License for more details.
2521+#
2522+# You should have received a copy of the GNU Lesser General Public License
2523+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2524+
2525+import os
2526+import yaml
2527+
2528+from charmhelpers.core import hookenv
2529+from charmhelpers.core import host
2530+from charmhelpers.core import templating
2531+
2532+from charmhelpers.core.services.base import ManagerCallback
2533+
2534+
2535+__all__ = ['RelationContext', 'TemplateCallback',
2536+ 'render_template', 'template']
2537+
2538+
2539+class RelationContext(dict):
2540+ """
2541+ Base class for a context generator that gets relation data from juju.
2542+
2543+ Subclasses must provide the attributes `name`, which is the name of the
2544+ interface of interest, `interface`, which is the type of the interface of
2545+ interest, and `required_keys`, which is the set of keys required for the
2546+ relation to be considered complete. The data for all interfaces matching
2547+ the `name` attribute that are complete will used to populate the dictionary
2548+ values (see `get_data`, below).
2549+
2550+ The generated context will be namespaced under the relation :attr:`name`,
2551+ to prevent potential naming conflicts.
2552+
2553+ :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
2554+ :param list additional_required_keys: Extend the list of :attr:`required_keys`
2555+ """
2556+ name = None
2557+ interface = None
2558+
2559+ def __init__(self, name=None, additional_required_keys=None):
2560+ if not hasattr(self, 'required_keys'):
2561+ self.required_keys = []
2562+
2563+ if name is not None:
2564+ self.name = name
2565+ if additional_required_keys:
2566+ self.required_keys.extend(additional_required_keys)
2567+ self.get_data()
2568+
2569+ def __bool__(self):
2570+ """
2571+ Returns True if all of the required_keys are available.
2572+ """
2573+ return self.is_ready()
2574+
2575+ __nonzero__ = __bool__
2576+
2577+ def __repr__(self):
2578+ return super(RelationContext, self).__repr__()
2579+
2580+ def is_ready(self):
2581+ """
2582+ Returns True if all of the `required_keys` are available from any units.
2583+ """
2584+ ready = len(self.get(self.name, [])) > 0
2585+ if not ready:
2586+ hookenv.log('Incomplete relation: {}'.format(self.__class__.__name__), hookenv.DEBUG)
2587+ return ready
2588+
2589+ def _is_ready(self, unit_data):
2590+ """
2591+ Helper method that tests a set of relation data and returns True if
2592+ all of the `required_keys` are present.
2593+ """
2594+ return set(unit_data.keys()).issuperset(set(self.required_keys))
2595+
2596+ def get_data(self):
2597+ """
2598+ Retrieve the relation data for each unit involved in a relation and,
2599+ if complete, store it in a list under `self[self.name]`. This
2600+ is automatically called when the RelationContext is instantiated.
2601+
2602+ The units are sorted lexographically first by the service ID, then by
2603+ the unit ID. Thus, if an interface has two other services, 'db:1'
2604+ and 'db:2', with 'db:1' having two units, 'wordpress/0' and 'wordpress/1',
2605+ and 'db:2' having one unit, 'mediawiki/0', all of which have a complete
2606+ set of data, the relation data for the units will be stored in the
2607+ order: 'wordpress/0', 'wordpress/1', 'mediawiki/0'.
2608+
2609+ If you only care about a single unit on the relation, you can just
2610+ access it as `{{ interface[0]['key'] }}`. However, if you can at all
2611+ support multiple units on a relation, you should iterate over the list,
2612+ like::
2613+
2614+ {% for unit in interface -%}
2615+ {{ unit['key'] }}{% if not loop.last %},{% endif %}
2616+ {%- endfor %}
2617+
2618+ Note that since all sets of relation data from all related services and
2619+ units are in a single list, if you need to know which service or unit a
2620+ set of data came from, you'll need to extend this class to preserve
2621+ that information.
2622+ """
2623+ if not hookenv.relation_ids(self.name):
2624+ return
2625+
2626+ ns = self.setdefault(self.name, [])
2627+ for rid in sorted(hookenv.relation_ids(self.name)):
2628+ for unit in sorted(hookenv.related_units(rid)):
2629+ reldata = hookenv.relation_get(rid=rid, unit=unit)
2630+ if self._is_ready(reldata):
2631+ ns.append(reldata)
2632+
2633+ def provide_data(self):
2634+ """
2635+ Return data to be relation_set for this interface.
2636+ """
2637+ return {}
2638+
2639+
2640+class MysqlRelation(RelationContext):
2641+ """
2642+ Relation context for the `mysql` interface.
2643+
2644+ :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
2645+ :param list additional_required_keys: Extend the list of :attr:`required_keys`
2646+ """
2647+ name = 'db'
2648+ interface = 'mysql'
2649+
2650+ def __init__(self, *args, **kwargs):
2651+ self.required_keys = ['host', 'user', 'password', 'database']
2652+ RelationContext.__init__(self, *args, **kwargs)
2653+
2654+
2655+class HttpRelation(RelationContext):
2656+ """
2657+ Relation context for the `http` interface.
2658+
2659+ :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
2660+ :param list additional_required_keys: Extend the list of :attr:`required_keys`
2661+ """
2662+ name = 'website'
2663+ interface = 'http'
2664+
2665+ def __init__(self, *args, **kwargs):
2666+ self.required_keys = ['host', 'port']
2667+ RelationContext.__init__(self, *args, **kwargs)
2668+
2669+ def provide_data(self):
2670+ return {
2671+ 'host': hookenv.unit_get('private-address'),
2672+ 'port': 80,
2673+ }
2674+
2675+
2676+class RequiredConfig(dict):
2677+ """
2678+ Data context that loads config options with one or more mandatory options.
2679+
2680+ Once the required options have been changed from their default values, all
2681+ config options will be available, namespaced under `config` to prevent
2682+ potential naming conflicts (for example, between a config option and a
2683+ relation property).
2684+
2685+ :param list *args: List of options that must be changed from their default values.
2686+ """
2687+
2688+ def __init__(self, *args):
2689+ self.required_options = args
2690+ self['config'] = hookenv.config()
2691+ with open(os.path.join(hookenv.charm_dir(), 'config.yaml')) as fp:
2692+ self.config = yaml.load(fp).get('options', {})
2693+
2694+ def __bool__(self):
2695+ for option in self.required_options:
2696+ if option not in self['config']:
2697+ return False
2698+ current_value = self['config'][option]
2699+ default_value = self.config[option].get('default')
2700+ if current_value == default_value:
2701+ return False
2702+ if current_value in (None, '') and default_value in (None, ''):
2703+ return False
2704+ return True
2705+
2706+ def __nonzero__(self):
2707+ return self.__bool__()
2708+
2709+
2710+class StoredContext(dict):
2711+ """
2712+ A data context that always returns the data that it was first created with.
2713+
2714+ This is useful to do a one-time generation of things like passwords, that
2715+ will thereafter use the same value that was originally generated, instead
2716+ of generating a new value each time it is run.
2717+ """
2718+ def __init__(self, file_name, config_data):
2719+ """
2720+ If the file exists, populate `self` with the data from the file.
2721+ Otherwise, populate with the given data and persist it to the file.
2722+ """
2723+ if os.path.exists(file_name):
2724+ self.update(self.read_context(file_name))
2725+ else:
2726+ self.store_context(file_name, config_data)
2727+ self.update(config_data)
2728+
2729+ def store_context(self, file_name, config_data):
2730+ if not os.path.isabs(file_name):
2731+ file_name = os.path.join(hookenv.charm_dir(), file_name)
2732+ with open(file_name, 'w') as file_stream:
2733+ os.fchmod(file_stream.fileno(), 0o600)
2734+ yaml.dump(config_data, file_stream)
2735+
2736+ def read_context(self, file_name):
2737+ if not os.path.isabs(file_name):
2738+ file_name = os.path.join(hookenv.charm_dir(), file_name)
2739+ with open(file_name, 'r') as file_stream:
2740+ data = yaml.load(file_stream)
2741+ if not data:
2742+ raise OSError("%s is empty" % file_name)
2743+ return data
2744+
2745+
2746+class TemplateCallback(ManagerCallback):
2747+ """
2748+ Callback class that will render a Jinja2 template, for use as a ready
2749+ action.
2750+
2751+ :param str source: The template source file, relative to
2752+ `$CHARM_DIR/templates`
2753+
2754+ :param str target: The target to write the rendered template to (or None)
2755+ :param str owner: The owner of the rendered file
2756+ :param str group: The group of the rendered file
2757+ :param int perms: The permissions of the rendered file
2758+ :param partial on_change_action: functools partial to be executed when
2759+ rendered file changes
2760+ :param jinja2 loader template_loader: A jinja2 template loader
2761+
2762+ :return str: The rendered template
2763+ """
2764+ def __init__(self, source, target,
2765+ owner='root', group='root', perms=0o444,
2766+ on_change_action=None, template_loader=None):
2767+ self.source = source
2768+ self.target = target
2769+ self.owner = owner
2770+ self.group = group
2771+ self.perms = perms
2772+ self.on_change_action = on_change_action
2773+ self.template_loader = template_loader
2774+
2775+ def __call__(self, manager, service_name, event_name):
2776+ pre_checksum = ''
2777+ if self.on_change_action and os.path.isfile(self.target):
2778+ pre_checksum = host.file_hash(self.target)
2779+ service = manager.get_service(service_name)
2780+ context = {'ctx': {}}
2781+ for ctx in service.get('required_data', []):
2782+ context.update(ctx)
2783+ context['ctx'].update(ctx)
2784+
2785+ result = templating.render(self.source, self.target, context,
2786+ self.owner, self.group, self.perms,
2787+ template_loader=self.template_loader)
2788+ if self.on_change_action:
2789+ if pre_checksum == host.file_hash(self.target):
2790+ hookenv.log(
2791+ 'No change detected: {}'.format(self.target),
2792+ hookenv.DEBUG)
2793+ else:
2794+ self.on_change_action()
2795+
2796+ return result
2797+
2798+
2799+# Convenience aliases for templates
2800+render_template = template = TemplateCallback
2801
2802=== added file 'hooks/charmhelpers/core/strutils.py'
2803--- hooks/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000
2804+++ hooks/charmhelpers/core/strutils.py 2015-12-17 05:44:32 +0000
2805@@ -0,0 +1,72 @@
2806+#!/usr/bin/env python
2807+# -*- coding: utf-8 -*-
2808+
2809+# Copyright 2014-2015 Canonical Limited.
2810+#
2811+# This file is part of charm-helpers.
2812+#
2813+# charm-helpers is free software: you can redistribute it and/or modify
2814+# it under the terms of the GNU Lesser General Public License version 3 as
2815+# published by the Free Software Foundation.
2816+#
2817+# charm-helpers is distributed in the hope that it will be useful,
2818+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2819+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2820+# GNU Lesser General Public License for more details.
2821+#
2822+# You should have received a copy of the GNU Lesser General Public License
2823+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2824+
2825+import six
2826+import re
2827+
2828+
2829+def bool_from_string(value):
2830+ """Interpret string value as boolean.
2831+
2832+ Returns True if value translates to True otherwise False.
2833+ """
2834+ if isinstance(value, six.string_types):
2835+ value = six.text_type(value)
2836+ else:
2837+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
2838+ raise ValueError(msg)
2839+
2840+ value = value.strip().lower()
2841+
2842+ if value in ['y', 'yes', 'true', 't', 'on']:
2843+ return True
2844+ elif value in ['n', 'no', 'false', 'f', 'off']:
2845+ return False
2846+
2847+ msg = "Unable to interpret string value '%s' as boolean" % (value)
2848+ raise ValueError(msg)
2849+
2850+
2851+def bytes_from_string(value):
2852+ """Interpret human readable string value as bytes.
2853+
2854+ Returns int
2855+ """
2856+ BYTE_POWER = {
2857+ 'K': 1,
2858+ 'KB': 1,
2859+ 'M': 2,
2860+ 'MB': 2,
2861+ 'G': 3,
2862+ 'GB': 3,
2863+ 'T': 4,
2864+ 'TB': 4,
2865+ 'P': 5,
2866+ 'PB': 5,
2867+ }
2868+ if isinstance(value, six.string_types):
2869+ value = six.text_type(value)
2870+ else:
2871+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
2872+ raise ValueError(msg)
2873+ matches = re.match("([0-9]+)([a-zA-Z]+)", value)
2874+ if not matches:
2875+ msg = "Unable to interpret string value '%s' as bytes" % (value)
2876+ raise ValueError(msg)
2877+ return int(matches.group(1)) * (1024 ** BYTE_POWER[matches.group(2)])
2878
2879=== added file 'hooks/charmhelpers/core/sysctl.py'
2880--- hooks/charmhelpers/core/sysctl.py 1970-01-01 00:00:00 +0000
2881+++ hooks/charmhelpers/core/sysctl.py 2015-12-17 05:44:32 +0000
2882@@ -0,0 +1,56 @@
2883+#!/usr/bin/env python
2884+# -*- coding: utf-8 -*-
2885+
2886+# Copyright 2014-2015 Canonical Limited.
2887+#
2888+# This file is part of charm-helpers.
2889+#
2890+# charm-helpers is free software: you can redistribute it and/or modify
2891+# it under the terms of the GNU Lesser General Public License version 3 as
2892+# published by the Free Software Foundation.
2893+#
2894+# charm-helpers is distributed in the hope that it will be useful,
2895+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2896+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2897+# GNU Lesser General Public License for more details.
2898+#
2899+# You should have received a copy of the GNU Lesser General Public License
2900+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2901+
2902+import yaml
2903+
2904+from subprocess import check_call
2905+
2906+from charmhelpers.core.hookenv import (
2907+ log,
2908+ DEBUG,
2909+ ERROR,
2910+)
2911+
2912+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
2913+
2914+
2915+def create(sysctl_dict, sysctl_file):
2916+ """Creates a sysctl.conf file from a YAML associative array
2917+
2918+ :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
2919+ :type sysctl_dict: str
2920+ :param sysctl_file: path to the sysctl file to be saved
2921+ :type sysctl_file: str or unicode
2922+ :returns: None
2923+ """
2924+ try:
2925+ sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
2926+ except yaml.YAMLError:
2927+ log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
2928+ level=ERROR)
2929+ return
2930+
2931+ with open(sysctl_file, "w") as fd:
2932+ for key, value in sysctl_dict_parsed.items():
2933+ fd.write("{}={}\n".format(key, value))
2934+
2935+ log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
2936+ level=DEBUG)
2937+
2938+ check_call(["sysctl", "-p", sysctl_file])
2939
2940=== added file 'hooks/charmhelpers/core/templating.py'
2941--- hooks/charmhelpers/core/templating.py 1970-01-01 00:00:00 +0000
2942+++ hooks/charmhelpers/core/templating.py 2015-12-17 05:44:32 +0000
2943@@ -0,0 +1,81 @@
2944+# Copyright 2014-2015 Canonical Limited.
2945+#
2946+# This file is part of charm-helpers.
2947+#
2948+# charm-helpers is free software: you can redistribute it and/or modify
2949+# it under the terms of the GNU Lesser General Public License version 3 as
2950+# published by the Free Software Foundation.
2951+#
2952+# charm-helpers is distributed in the hope that it will be useful,
2953+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2954+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2955+# GNU Lesser General Public License for more details.
2956+#
2957+# You should have received a copy of the GNU Lesser General Public License
2958+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2959+
2960+import os
2961+
2962+from charmhelpers.core import host
2963+from charmhelpers.core import hookenv
2964+
2965+
2966+def render(source, target, context, owner='root', group='root',
2967+ perms=0o444, templates_dir=None, encoding='UTF-8', template_loader=None):
2968+ """
2969+ Render a template.
2970+
2971+ The `source` path, if not absolute, is relative to the `templates_dir`.
2972+
2973+ The `target` path should be absolute. It can also be `None`, in which
2974+ case no file will be written.
2975+
2976+ The context should be a dict containing the values to be replaced in the
2977+ template.
2978+
2979+ The `owner`, `group`, and `perms` options will be passed to `write_file`.
2980+
2981+ If omitted, `templates_dir` defaults to the `templates` folder in the charm.
2982+
2983+ The rendered template will be written to the file as well as being returned
2984+ as a string.
2985+
2986+ Note: Using this requires python-jinja2; if it is not installed, calling
2987+ this will attempt to use charmhelpers.fetch.apt_install to install it.
2988+ """
2989+ try:
2990+ from jinja2 import FileSystemLoader, Environment, exceptions
2991+ except ImportError:
2992+ try:
2993+ from charmhelpers.fetch import apt_install
2994+ except ImportError:
2995+ hookenv.log('Could not import jinja2, and could not import '
2996+ 'charmhelpers.fetch to install it',
2997+ level=hookenv.ERROR)
2998+ raise
2999+ apt_install('python-jinja2', fatal=True)
3000+ from jinja2 import FileSystemLoader, Environment, exceptions
3001+
3002+ if template_loader:
3003+ template_env = Environment(loader=template_loader)
3004+ else:
3005+ if templates_dir is None:
3006+ templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
3007+ template_env = Environment(loader=FileSystemLoader(templates_dir))
3008+ try:
3009+ source = source
3010+ template = template_env.get_template(source)
3011+ except exceptions.TemplateNotFound as e:
3012+ hookenv.log('Could not load template %s from %s.' %
3013+ (source, templates_dir),
3014+ level=hookenv.ERROR)
3015+ raise e
3016+ content = template.render(context)
3017+ if target is not None:
3018+ target_dir = os.path.dirname(target)
3019+ if not os.path.exists(target_dir):
3020+ # This is a terrible default directory permission, as the file
3021+ # or its siblings will often contain secrets.
3022+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
3023+ host.write_file(target, content.encode(encoding), owner, group, perms)
3024+ return content
3025
3026=== added file 'hooks/charmhelpers/core/unitdata.py'
3027--- hooks/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000
3028+++ hooks/charmhelpers/core/unitdata.py 2015-12-17 05:44:32 +0000
3029@@ -0,0 +1,521 @@
3030+#!/usr/bin/env python
3031+# -*- coding: utf-8 -*-
3032+#
3033+# Copyright 2014-2015 Canonical Limited.
3034+#
3035+# This file is part of charm-helpers.
3036+#
3037+# charm-helpers is free software: you can redistribute it and/or modify
3038+# it under the terms of the GNU Lesser General Public License version 3 as
3039+# published by the Free Software Foundation.
3040+#
3041+# charm-helpers is distributed in the hope that it will be useful,
3042+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3043+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3044+# GNU Lesser General Public License for more details.
3045+#
3046+# You should have received a copy of the GNU Lesser General Public License
3047+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3048+#
3049+#
3050+# Authors:
3051+# Kapil Thangavelu <kapil.foss@gmail.com>
3052+#
3053+"""
3054+Intro
3055+-----
3056+
3057+A simple way to store state in units. This provides a key value
3058+storage with support for versioned, transactional operation,
3059+and can calculate deltas from previous values to simplify unit logic
3060+when processing changes.
3061+
3062+
3063+Hook Integration
3064+----------------
3065+
3066+There are several extant frameworks for hook execution, including
3067+
3068+ - charmhelpers.core.hookenv.Hooks
3069+ - charmhelpers.core.services.ServiceManager
3070+
3071+The storage classes are framework agnostic, one simple integration is
3072+via the HookData contextmanager. It will record the current hook
3073+execution environment (including relation data, config data, etc.),
3074+setup a transaction and allow easy access to the changes from
3075+previously seen values. One consequence of the integration is the
3076+reservation of particular keys ('rels', 'unit', 'env', 'config',
3077+'charm_revisions') for their respective values.
3078+
3079+Here's a fully worked integration example using hookenv.Hooks::
3080+
3081+ from charmhelper.core import hookenv, unitdata
3082+
3083+ hook_data = unitdata.HookData()
3084+ db = unitdata.kv()
3085+ hooks = hookenv.Hooks()
3086+
3087+ @hooks.hook
3088+ def config_changed():
3089+ # Print all changes to configuration from previously seen
3090+ # values.
3091+ for changed, (prev, cur) in hook_data.conf.items():
3092+ print('config changed', changed,
3093+ 'previous value', prev,
3094+ 'current value', cur)
3095+
3096+ # Get some unit specific bookeeping
3097+ if not db.get('pkg_key'):
3098+ key = urllib.urlopen('https://example.com/pkg_key').read()
3099+ db.set('pkg_key', key)
3100+
3101+ # Directly access all charm config as a mapping.
3102+ conf = db.getrange('config', True)
3103+
3104+ # Directly access all relation data as a mapping
3105+ rels = db.getrange('rels', True)
3106+
3107+ if __name__ == '__main__':
3108+ with hook_data():
3109+ hook.execute()
3110+
3111+
3112+A more basic integration is via the hook_scope context manager which simply
3113+manages transaction scope (and records hook name, and timestamp)::
3114+
3115+ >>> from unitdata import kv
3116+ >>> db = kv()
3117+ >>> with db.hook_scope('install'):
3118+ ... # do work, in transactional scope.
3119+ ... db.set('x', 1)
3120+ >>> db.get('x')
3121+ 1
3122+
3123+
3124+Usage
3125+-----
3126+
3127+Values are automatically json de/serialized to preserve basic typing
3128+and complex data struct capabilities (dicts, lists, ints, booleans, etc).
3129+
3130+Individual values can be manipulated via get/set::
3131+
3132+ >>> kv.set('y', True)
3133+ >>> kv.get('y')
3134+ True
3135+
3136+ # We can set complex values (dicts, lists) as a single key.
3137+ >>> kv.set('config', {'a': 1, 'b': True'})
3138+
3139+ # Also supports returning dictionaries as a record which
3140+ # provides attribute access.
3141+ >>> config = kv.get('config', record=True)
3142+ >>> config.b
3143+ True
3144+
3145+
3146+Groups of keys can be manipulated with update/getrange::
3147+
3148+ >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
3149+ >>> kv.getrange('gui.', strip=True)
3150+ {'z': 1, 'y': 2}
3151+
3152+When updating values, its very helpful to understand which values
3153+have actually changed and how have they changed. The storage
3154+provides a delta method to provide for this::
3155+
3156+ >>> data = {'debug': True, 'option': 2}
3157+ >>> delta = kv.delta(data, 'config.')
3158+ >>> delta.debug.previous
3159+ None
3160+ >>> delta.debug.current
3161+ True
3162+ >>> delta
3163+ {'debug': (None, True), 'option': (None, 2)}
3164+
3165+Note the delta method does not persist the actual change, it needs to
3166+be explicitly saved via 'update' method::
3167+
3168+ >>> kv.update(data, 'config.')
3169+
3170+Values modified in the context of a hook scope retain historical values
3171+associated to the hookname.
3172+
3173+ >>> with db.hook_scope('config-changed'):
3174+ ... db.set('x', 42)
3175+ >>> db.gethistory('x')
3176+ [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
3177+ (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
3178+
3179+"""
3180+
3181+import collections
3182+import contextlib
3183+import datetime
3184+import itertools
3185+import json
3186+import os
3187+import pprint
3188+import sqlite3
3189+import sys
3190+
3191+__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
3192+
3193+
3194+class Storage(object):
3195+ """Simple key value database for local unit state within charms.
3196+
3197+ Modifications are not persisted unless :meth:`flush` is called.
3198+
3199+ To support dicts, lists, integer, floats, and booleans values
3200+ are automatically json encoded/decoded.
3201+ """
3202+ def __init__(self, path=None):
3203+ self.db_path = path
3204+ if path is None:
3205+ if 'UNIT_STATE_DB' in os.environ:
3206+ self.db_path = os.environ['UNIT_STATE_DB']
3207+ else:
3208+ self.db_path = os.path.join(
3209+ os.environ.get('CHARM_DIR', ''), '.unit-state.db')
3210+ self.conn = sqlite3.connect('%s' % self.db_path)
3211+ self.cursor = self.conn.cursor()
3212+ self.revision = None
3213+ self._closed = False
3214+ self._init()
3215+
3216+ def close(self):
3217+ if self._closed:
3218+ return
3219+ self.flush(False)
3220+ self.cursor.close()
3221+ self.conn.close()
3222+ self._closed = True
3223+
3224+ def get(self, key, default=None, record=False):
3225+ self.cursor.execute('select data from kv where key=?', [key])
3226+ result = self.cursor.fetchone()
3227+ if not result:
3228+ return default
3229+ if record:
3230+ return Record(json.loads(result[0]))
3231+ return json.loads(result[0])
3232+
3233+ def getrange(self, key_prefix, strip=False):
3234+ """
3235+ Get a range of keys starting with a common prefix as a mapping of
3236+ keys to values.
3237+
3238+ :param str key_prefix: Common prefix among all keys
3239+ :param bool strip: Optionally strip the common prefix from the key
3240+ names in the returned dict
3241+ :return dict: A (possibly empty) dict of key-value mappings
3242+ """
3243+ self.cursor.execute("select key, data from kv where key like ?",
3244+ ['%s%%' % key_prefix])
3245+ result = self.cursor.fetchall()
3246+
3247+ if not result:
3248+ return {}
3249+ if not strip:
3250+ key_prefix = ''
3251+ return dict([
3252+ (k[len(key_prefix):], json.loads(v)) for k, v in result])
3253+
3254+ def update(self, mapping, prefix=""):
3255+ """
3256+ Set the values of multiple keys at once.
3257+
3258+ :param dict mapping: Mapping of keys to values
3259+ :param str prefix: Optional prefix to apply to all keys in `mapping`
3260+ before setting
3261+ """
3262+ for k, v in mapping.items():
3263+ self.set("%s%s" % (prefix, k), v)
3264+
3265+ def unset(self, key):
3266+ """
3267+ Remove a key from the database entirely.
3268+ """
3269+ self.cursor.execute('delete from kv where key=?', [key])
3270+ if self.revision and self.cursor.rowcount:
3271+ self.cursor.execute(
3272+ 'insert into kv_revisions values (?, ?, ?)',
3273+ [key, self.revision, json.dumps('DELETED')])
3274+
3275+ def unsetrange(self, keys=None, prefix=""):
3276+ """
3277+ Remove a range of keys starting with a common prefix, from the database
3278+ entirely.
3279+
3280+ :param list keys: List of keys to remove.
3281+ :param str prefix: Optional prefix to apply to all keys in ``keys``
3282+ before removing.
3283+ """
3284+ if keys is not None:
3285+ keys = ['%s%s' % (prefix, key) for key in keys]
3286+ self.cursor.execute('delete from kv where key in (%s)' % ','.join(['?'] * len(keys)), keys)
3287+ if self.revision and self.cursor.rowcount:
3288+ self.cursor.execute(
3289+ 'insert into kv_revisions values %s' % ','.join(['(?, ?, ?)'] * len(keys)),
3290+ list(itertools.chain.from_iterable((key, self.revision, json.dumps('DELETED')) for key in keys)))
3291+ else:
3292+ self.cursor.execute('delete from kv where key like ?',
3293+ ['%s%%' % prefix])
3294+ if self.revision and self.cursor.rowcount:
3295+ self.cursor.execute(
3296+ 'insert into kv_revisions values (?, ?, ?)',
3297+ ['%s%%' % prefix, self.revision, json.dumps('DELETED')])
3298+
3299+ def set(self, key, value):
3300+ """
3301+ Set a value in the database.
3302+
3303+ :param str key: Key to set the value for
3304+ :param value: Any JSON-serializable value to be set
3305+ """
3306+ serialized = json.dumps(value)
3307+
3308+ self.cursor.execute('select data from kv where key=?', [key])
3309+ exists = self.cursor.fetchone()
3310+
3311+ # Skip mutations to the same value
3312+ if exists:
3313+ if exists[0] == serialized:
3314+ return value
3315+
3316+ if not exists:
3317+ self.cursor.execute(
3318+ 'insert into kv (key, data) values (?, ?)',
3319+ (key, serialized))
3320+ else:
3321+ self.cursor.execute('''
3322+ update kv
3323+ set data = ?
3324+ where key = ?''', [serialized, key])
3325+
3326+ # Save
3327+ if not self.revision:
3328+ return value
3329+
3330+ self.cursor.execute(
3331+ 'select 1 from kv_revisions where key=? and revision=?',
3332+ [key, self.revision])
3333+ exists = self.cursor.fetchone()
3334+
3335+ if not exists:
3336+ self.cursor.execute(
3337+ '''insert into kv_revisions (
3338+ revision, key, data) values (?, ?, ?)''',
3339+ (self.revision, key, serialized))
3340+ else:
3341+ self.cursor.execute(
3342+ '''
3343+ update kv_revisions
3344+ set data = ?
3345+ where key = ?
3346+ and revision = ?''',
3347+ [serialized, key, self.revision])
3348+
3349+ return value
3350+
3351+ def delta(self, mapping, prefix):
3352+ """
3353+ return a delta containing values that have changed.
3354+ """
3355+ previous = self.getrange(prefix, strip=True)
3356+ if not previous:
3357+ pk = set()
3358+ else:
3359+ pk = set(previous.keys())
3360+ ck = set(mapping.keys())
3361+ delta = DeltaSet()
3362+
3363+ # added
3364+ for k in ck.difference(pk):
3365+ delta[k] = Delta(None, mapping[k])
3366+
3367+ # removed
3368+ for k in pk.difference(ck):
3369+ delta[k] = Delta(previous[k], None)
3370+
3371+ # changed
3372+ for k in pk.intersection(ck):
3373+ c = mapping[k]
3374+ p = previous[k]
3375+ if c != p:
3376+ delta[k] = Delta(p, c)
3377+
3378+ return delta
3379+
3380+ @contextlib.contextmanager
3381+ def hook_scope(self, name=""):
3382+ """Scope all future interactions to the current hook execution
3383+ revision."""
3384+ assert not self.revision
3385+ self.cursor.execute(
3386+ 'insert into hooks (hook, date) values (?, ?)',
3387+ (name or sys.argv[0],
3388+ datetime.datetime.utcnow().isoformat()))
3389+ self.revision = self.cursor.lastrowid
3390+ try:
3391+ yield self.revision
3392+ self.revision = None
3393+ except:
3394+ self.flush(False)
3395+ self.revision = None
3396+ raise
3397+ else:
3398+ self.flush()
3399+
3400+ def flush(self, save=True):
3401+ if save:
3402+ self.conn.commit()
3403+ elif self._closed:
3404+ return
3405+ else:
3406+ self.conn.rollback()
3407+
3408+ def _init(self):
3409+ self.cursor.execute('''
3410+ create table if not exists kv (
3411+ key text,
3412+ data text,
3413+ primary key (key)
3414+ )''')
3415+ self.cursor.execute('''
3416+ create table if not exists kv_revisions (
3417+ key text,
3418+ revision integer,
3419+ data text,
3420+ primary key (key, revision)
3421+ )''')
3422+ self.cursor.execute('''
3423+ create table if not exists hooks (
3424+ version integer primary key autoincrement,
3425+ hook text,
3426+ date text
3427+ )''')
3428+ self.conn.commit()
3429+
3430+ def gethistory(self, key, deserialize=False):
3431+ self.cursor.execute(
3432+ '''
3433+ select kv.revision, kv.key, kv.data, h.hook, h.date
3434+ from kv_revisions kv,
3435+ hooks h
3436+ where kv.key=?
3437+ and kv.revision = h.version
3438+ ''', [key])
3439+ if deserialize is False:
3440+ return self.cursor.fetchall()
3441+ return map(_parse_history, self.cursor.fetchall())
3442+
3443+ def debug(self, fh=sys.stderr):
3444+ self.cursor.execute('select * from kv')
3445+ pprint.pprint(self.cursor.fetchall(), stream=fh)
3446+ self.cursor.execute('select * from kv_revisions')
3447+ pprint.pprint(self.cursor.fetchall(), stream=fh)
3448+
3449+
3450+def _parse_history(d):
3451+ return (d[0], d[1], json.loads(d[2]), d[3],
3452+ datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
3453+
3454+
3455+class HookData(object):
3456+ """Simple integration for existing hook exec frameworks.
3457+
3458+ Records all unit information, and stores deltas for processing
3459+ by the hook.
3460+
3461+ Sample::
3462+
3463+ from charmhelper.core import hookenv, unitdata
3464+
3465+ changes = unitdata.HookData()
3466+ db = unitdata.kv()
3467+ hooks = hookenv.Hooks()
3468+
3469+ @hooks.hook
3470+ def config_changed():
3471+ # View all changes to configuration
3472+ for changed, (prev, cur) in changes.conf.items():
3473+ print('config changed', changed,
3474+ 'previous value', prev,
3475+ 'current value', cur)
3476+
3477+ # Get some unit specific bookeeping
3478+ if not db.get('pkg_key'):
3479+ key = urllib.urlopen('https://example.com/pkg_key').read()
3480+ db.set('pkg_key', key)
3481+
3482+ if __name__ == '__main__':
3483+ with changes():
3484+ hook.execute()
3485+
3486+ """
3487+ def __init__(self):
3488+ self.kv = kv()
3489+ self.conf = None
3490+ self.rels = None
3491+
3492+ @contextlib.contextmanager
3493+ def __call__(self):
3494+ from charmhelpers.core import hookenv
3495+ hook_name = hookenv.hook_name()
3496+
3497+ with self.kv.hook_scope(hook_name):
3498+ self._record_charm_version(hookenv.charm_dir())
3499+ delta_config, delta_relation = self._record_hook(hookenv)
3500+ yield self.kv, delta_config, delta_relation
3501+
3502+ def _record_charm_version(self, charm_dir):
3503+ # Record revisions.. charm revisions are meaningless
3504+ # to charm authors as they don't control the revision.
3505+ # so logic dependnent on revision is not particularly
3506+ # useful, however it is useful for debugging analysis.
3507+ charm_rev = open(
3508+ os.path.join(charm_dir, 'revision')).read().strip()
3509+ charm_rev = charm_rev or '0'
3510+ revs = self.kv.get('charm_revisions', [])
3511+ if charm_rev not in revs:
3512+ revs.append(charm_rev.strip() or '0')
3513+ self.kv.set('charm_revisions', revs)
3514+
3515+ def _record_hook(self, hookenv):
3516+ data = hookenv.execution_environment()
3517+ self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
3518+ self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
3519+ self.kv.set('env', dict(data['env']))
3520+ self.kv.set('unit', data['unit'])
3521+ self.kv.set('relid', data.get('relid'))
3522+ return conf_delta, rels_delta
3523+
3524+
3525+class Record(dict):
3526+
3527+ __slots__ = ()
3528+
3529+ def __getattr__(self, k):
3530+ if k in self:
3531+ return self[k]
3532+ raise AttributeError(k)
3533+
3534+
3535+class DeltaSet(Record):
3536+
3537+ __slots__ = ()
3538+
3539+
3540+Delta = collections.namedtuple('Delta', ['previous', 'current'])
3541+
3542+
3543+_KV = None
3544+
3545+
3546+def kv():
3547+ global _KV
3548+ if _KV is None:
3549+ _KV = Storage()
3550+ return _KV
3551
3552=== modified file 'hooks/hooks.py'
3553--- hooks/hooks.py 2015-12-06 22:48:27 +0000
3554+++ hooks/hooks.py 2015-12-17 05:44:32 +0000
3555@@ -15,6 +15,9 @@
3556 import glob
3557 import utils
3558
3559+from charmhelpers.core import hookenv
3560+from charmhelpers.core import unitdata
3561+
3562 ###############################################################################
3563 # Global variables
3564 ###############################################################################
3565@@ -22,9 +25,8 @@
3566 default_squid3_config = "%s/squid.conf" % default_squid3_config_dir
3567 default_squid3_config_cache_dir = "/var/run/squid3"
3568 hook_name = os.path.basename(sys.argv[0])
3569-HOOK_START = False
3570-HOOK_AUTH_HELPER_JOINED = False
3571-STATE_DELAYED_START = False
3572+db = unitdata.kv()
3573+db_changes = unitdata.HookData()
3574 ###############################################################################
3575 # Supporting functions
3576 ###############################################################################
3577@@ -321,24 +323,26 @@
3578 retVal = subprocess.call(
3579 ['/usr/sbin/squid3', '-f', squid3_config, '-k', 'parse'])
3580 if retVal == 1:
3581- return(False)
3582+ utils.juju_log('CRITICAL', 'Invalid squid configuration.')
3583+ return False
3584 elif retVal == 0:
3585- return(True)
3586+ return True
3587 else:
3588- return(False)
3589+ utils.juju_log('CRITICAL', 'Invalid squid configuration.')
3590+ return False
3591 elif action == 'status':
3592 status = subprocess.check_output(['status', 'squid3'])
3593 if re.search('running', status) is not None:
3594- return(True)
3595+ return True
3596 else:
3597- return(False)
3598+ return False
3599 elif action in ('start', 'stop', 'reload', 'restart'):
3600 utils.juju_log('INFO', 'Requesting %s of squid3 service.' % action)
3601 retVal = subprocess.call([action, 'squid3'])
3602 if retVal == 0:
3603- return(True)
3604+ return True
3605 else:
3606- return(False)
3607+ return False
3608
3609
3610 def update_nrpe_checks():
3611@@ -400,6 +404,22 @@
3612 return (utils.install_unattended('squid3', 'python-jinja2'))
3613
3614
3615+def service_can_start():
3616+ # If wait_for_auth_helper is set, wait until squid has started and
3617+ # the squid-auth-helper relation has been joined.
3618+ config_data = config_get()
3619+ if not config_data['wait_for_auth_helper']:
3620+ return True
3621+ if hookenv.is_relation_made('squid-auth-helper') and db.get('hook_start'):
3622+ utils.juju_log('INFO',
3623+ 'Squid auth helper available, squid may start...')
3624+ return True
3625+ else:
3626+ utils.juju_log('INFO', 'Squid not ready, waiting for auth helper...')
3627+ service_squid3('stop')
3628+ return False
3629+
3630+
3631 def config_changed():
3632 current_service_ports = get_service_ports()
3633 construct_squid3_config()
3634@@ -408,11 +428,7 @@
3635 updated_service_ports = get_service_ports()
3636 update_service_ports(current_service_ports, updated_service_ports)
3637
3638- config_data = config_get()
3639- if config_data['wait_for_auth_helper'] and not STATE_DELAYED_START:
3640- # unable to parse squid3 configuration without auth helper in
3641- # place.
3642- utils.juju_log('INFO', 'Squid not started, waiting for auth helper...')
3643+ if not service_can_start():
3644 return
3645
3646 if service_squid3('check'):
3647@@ -424,36 +440,26 @@
3648 sys.exit(1)
3649
3650
3651-def start_hook(start=None, auth_helper=None):
3652- global HOOK_START
3653- global HOOK_AUTH_HELPER_JOINED
3654-
3655+def start_hook(start=None):
3656 if start:
3657- HOOK_START = True
3658- if auth_helper:
3659- HOOK_AUTH_HELPER_JOINED = True
3660-
3661- config_data = config_get()
3662- if config_data['wait_for_auth_helper']:
3663- if HOOK_AUTH_HELPER_JOINED and HOOK_START:
3664- utils.juju_log('INFO', 'Squid auth helper available, starting...')
3665- if service_squid3('check'):
3666- STATE_DELAYED_START = True
3667- else:
3668- sys.exit(1)
3669- else:
3670- utils.juju_log('INFO', 'Waiting for auth helper...')
3671- return
3672-
3673- if service_squid3("status"):
3674- return(service_squid3("restart"))
3675- else:
3676- return(service_squid3("start"))
3677+ db.set('hook_start', True)
3678+
3679+ if service_can_start():
3680+ if not service_squid3('check'):
3681+ sys.exit(1)
3682+ else:
3683+ return
3684+
3685+ if service_squid3('status'):
3686+ return(service_squid3('restart'))
3687+ else:
3688+ return(service_squid3('start'))
3689
3690
3691 def stop_hook():
3692- if service_squid3("status"):
3693- return(service_squid3("stop"))
3694+ db.unset('hook_start')
3695+ if service_squid3('status'):
3696+ return(service_squid3('stop'))
3697
3698
3699 def proxy_interface(hook_name=None):
3700@@ -478,7 +484,7 @@
3701 elif hook_name == "stop":
3702 stop_hook()
3703 elif hook_name == "squid-auth-helper-relation-joined":
3704- start_hook(auth_helper=True)
3705+ start_hook()
3706 elif hook_name == "cached-website-relation-joined":
3707 proxy_interface("joined")
3708 elif hook_name == "cached-website-relation-changed":
3709@@ -498,4 +504,5 @@
3710 sys.exit(1)
3711
3712 if __name__ == '__main__':
3713- main()
3714+ with db_changes():
3715+ main()
3716
3717=== added directory 'scripts'
3718=== added file 'scripts/charm_helpers_sync.py'
3719--- scripts/charm_helpers_sync.py 1970-01-01 00:00:00 +0000
3720+++ scripts/charm_helpers_sync.py 2015-12-17 05:44:32 +0000
3721@@ -0,0 +1,253 @@
3722+#!/usr/bin/python
3723+
3724+# Copyright 2014-2015 Canonical Limited.
3725+#
3726+# This file is part of charm-helpers.
3727+#
3728+# charm-helpers is free software: you can redistribute it and/or modify
3729+# it under the terms of the GNU Lesser General Public License version 3 as
3730+# published by the Free Software Foundation.
3731+#
3732+# charm-helpers is distributed in the hope that it will be useful,
3733+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3734+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3735+# GNU Lesser General Public License for more details.
3736+#
3737+# You should have received a copy of the GNU Lesser General Public License
3738+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3739+
3740+# Authors:
3741+# Adam Gandelman <adamg@ubuntu.com>
3742+
3743+import logging
3744+import optparse
3745+import os
3746+import subprocess
3747+import shutil
3748+import sys
3749+import tempfile
3750+import yaml
3751+from fnmatch import fnmatch
3752+
3753+import six
3754+
3755+CHARM_HELPERS_BRANCH = 'lp:charm-helpers'
3756+
3757+
3758+def parse_config(conf_file):
3759+ if not os.path.isfile(conf_file):
3760+ logging.error('Invalid config file: %s.' % conf_file)
3761+ return False
3762+ return yaml.load(open(conf_file).read())
3763+
3764+
3765+def clone_helpers(work_dir, branch):
3766+ dest = os.path.join(work_dir, 'charm-helpers')
3767+ logging.info('Checking out %s to %s.' % (branch, dest))
3768+ cmd = ['bzr', 'checkout', '--lightweight', branch, dest]
3769+ subprocess.check_call(cmd)
3770+ return dest
3771+
3772+
3773+def _module_path(module):
3774+ return os.path.join(*module.split('.'))
3775+
3776+
3777+def _src_path(src, module):
3778+ return os.path.join(src, 'charmhelpers', _module_path(module))
3779+
3780+
3781+def _dest_path(dest, module):
3782+ return os.path.join(dest, _module_path(module))
3783+
3784+
3785+def _is_pyfile(path):
3786+ return os.path.isfile(path + '.py')
3787+
3788+
3789+def ensure_init(path):
3790+ '''
3791+ ensure directories leading up to path are importable, omitting
3792+ parent directory, eg path='/hooks/helpers/foo'/:
3793+ hooks/
3794+ hooks/helpers/__init__.py
3795+ hooks/helpers/foo/__init__.py
3796+ '''
3797+ for d, dirs, files in os.walk(os.path.join(*path.split('/')[:2])):
3798+ _i = os.path.join(d, '__init__.py')
3799+ if not os.path.exists(_i):
3800+ logging.info('Adding missing __init__.py: %s' % _i)
3801+ open(_i, 'wb').close()
3802+
3803+
3804+def sync_pyfile(src, dest):
3805+ src = src + '.py'
3806+ src_dir = os.path.dirname(src)
3807+ logging.info('Syncing pyfile: %s -> %s.' % (src, dest))
3808+ if not os.path.exists(dest):
3809+ os.makedirs(dest)
3810+ shutil.copy(src, dest)
3811+ if os.path.isfile(os.path.join(src_dir, '__init__.py')):
3812+ shutil.copy(os.path.join(src_dir, '__init__.py'),
3813+ dest)
3814+ ensure_init(dest)
3815+
3816+
3817+def get_filter(opts=None):
3818+ opts = opts or []
3819+ if 'inc=*' in opts:
3820+ # do not filter any files, include everything
3821+ return None
3822+
3823+ def _filter(dir, ls):
3824+ incs = [opt.split('=').pop() for opt in opts if 'inc=' in opt]
3825+ _filter = []
3826+ for f in ls:
3827+ _f = os.path.join(dir, f)
3828+
3829+ if not os.path.isdir(_f) and not _f.endswith('.py') and incs:
3830+ if True not in [fnmatch(_f, inc) for inc in incs]:
3831+ logging.debug('Not syncing %s, does not match include '
3832+ 'filters (%s)' % (_f, incs))
3833+ _filter.append(f)
3834+ else:
3835+ logging.debug('Including file, which matches include '
3836+ 'filters (%s): %s' % (incs, _f))
3837+ elif (os.path.isfile(_f) and not _f.endswith('.py')):
3838+ logging.debug('Not syncing file: %s' % f)
3839+ _filter.append(f)
3840+ elif (os.path.isdir(_f) and not
3841+ os.path.isfile(os.path.join(_f, '__init__.py'))):
3842+ logging.debug('Not syncing directory: %s' % f)
3843+ _filter.append(f)
3844+ return _filter
3845+ return _filter
3846+
3847+
3848+def sync_directory(src, dest, opts=None):
3849+ if os.path.exists(dest):
3850+ logging.debug('Removing existing directory: %s' % dest)
3851+ shutil.rmtree(dest)
3852+ logging.info('Syncing directory: %s -> %s.' % (src, dest))
3853+
3854+ shutil.copytree(src, dest, ignore=get_filter(opts))
3855+ ensure_init(dest)
3856+
3857+
3858+def sync(src, dest, module, opts=None):
3859+
3860+ # Sync charmhelpers/__init__.py for bootstrap code.
3861+ sync_pyfile(_src_path(src, '__init__'), dest)
3862+
3863+ # Sync other __init__.py files in the path leading to module.
3864+ m = []
3865+ steps = module.split('.')[:-1]
3866+ while steps:
3867+ m.append(steps.pop(0))
3868+ init = '.'.join(m + ['__init__'])
3869+ sync_pyfile(_src_path(src, init),
3870+ os.path.dirname(_dest_path(dest, init)))
3871+
3872+ # Sync the module, or maybe a .py file.
3873+ if os.path.isdir(_src_path(src, module)):
3874+ sync_directory(_src_path(src, module), _dest_path(dest, module), opts)
3875+ elif _is_pyfile(_src_path(src, module)):
3876+ sync_pyfile(_src_path(src, module),
3877+ os.path.dirname(_dest_path(dest, module)))
3878+ else:
3879+ logging.warn('Could not sync: %s. Neither a pyfile or directory, '
3880+ 'does it even exist?' % module)
3881+
3882+
3883+def parse_sync_options(options):
3884+ if not options:
3885+ return []
3886+ return options.split(',')
3887+
3888+
3889+def extract_options(inc, global_options=None):
3890+ global_options = global_options or []
3891+ if global_options and isinstance(global_options, six.string_types):
3892+ global_options = [global_options]
3893+ if '|' not in inc:
3894+ return (inc, global_options)
3895+ inc, opts = inc.split('|')
3896+ return (inc, parse_sync_options(opts) + global_options)
3897+
3898+
3899+def sync_helpers(include, src, dest, options=None):
3900+ if not os.path.isdir(dest):
3901+ os.makedirs(dest)
3902+
3903+ global_options = parse_sync_options(options)
3904+
3905+ for inc in include:
3906+ if isinstance(inc, str):
3907+ inc, opts = extract_options(inc, global_options)
3908+ sync(src, dest, inc, opts)
3909+ elif isinstance(inc, dict):
3910+ # could also do nested dicts here.
3911+ for k, v in six.iteritems(inc):
3912+ if isinstance(v, list):
3913+ for m in v:
3914+ inc, opts = extract_options(m, global_options)
3915+ sync(src, dest, '%s.%s' % (k, inc), opts)
3916+
3917+if __name__ == '__main__':
3918+ parser = optparse.OptionParser()
3919+ parser.add_option('-c', '--config', action='store', dest='config',
3920+ default=None, help='helper config file')
3921+ parser.add_option('-D', '--debug', action='store_true', dest='debug',
3922+ default=False, help='debug')
3923+ parser.add_option('-b', '--branch', action='store', dest='branch',
3924+ help='charm-helpers bzr branch (overrides config)')
3925+ parser.add_option('-d', '--destination', action='store', dest='dest_dir',
3926+ help='sync destination dir (overrides config)')
3927+ (opts, args) = parser.parse_args()
3928+
3929+ if opts.debug:
3930+ logging.basicConfig(level=logging.DEBUG)
3931+ else:
3932+ logging.basicConfig(level=logging.INFO)
3933+
3934+ if opts.config:
3935+ logging.info('Loading charm helper config from %s.' % opts.config)
3936+ config = parse_config(opts.config)
3937+ if not config:
3938+ logging.error('Could not parse config from %s.' % opts.config)
3939+ sys.exit(1)
3940+ else:
3941+ config = {}
3942+
3943+ if 'branch' not in config:
3944+ config['branch'] = CHARM_HELPERS_BRANCH
3945+ if opts.branch:
3946+ config['branch'] = opts.branch
3947+ if opts.dest_dir:
3948+ config['destination'] = opts.dest_dir
3949+
3950+ if 'destination' not in config:
3951+ logging.error('No destination dir. specified as option or config.')
3952+ sys.exit(1)
3953+
3954+ if 'include' not in config:
3955+ if not args:
3956+ logging.error('No modules to sync specified as option or config.')
3957+ sys.exit(1)
3958+ config['include'] = []
3959+ [config['include'].append(a) for a in args]
3960+
3961+ sync_options = None
3962+ if 'options' in config:
3963+ sync_options = config['options']
3964+ tmpd = tempfile.mkdtemp()
3965+ try:
3966+ checkout = clone_helpers(tmpd, config['branch'])
3967+ sync_helpers(config['include'], checkout, config['destination'],
3968+ options=sync_options)
3969+ except Exception as e:
3970+ logging.error("Could not sync: %s" % e)
3971+ raise e
3972+ finally:
3973+ logging.debug('Cleaning up %s' % tmpd)
3974+ shutil.rmtree(tmpd)

Subscribers

People subscribed via source and target branches

to all changes: