Merge ~stub/charms/+source/postgresql:reactive into ~stub/charms/+source/postgresql:master

Proposed by Stuart Bishop on 2016-01-12
Status: Rejected
Rejected by: Stuart Bishop on 2016-01-14
Proposed branch: ~stub/charms/+source/postgresql:reactive
Merge into: ~stub/charms/+source/postgresql:master
Diff against target: 24789 lines (+14743/-205)
106 files modified
.gitignore (+5/-0)
Makefile (+31/-3)
README.md (+1/-1)
charm-helpers.yaml (+4/-2)
config.yaml (+9/-15)
dev/null (+0/-129)
hooks/config-changed (+1/-0)
hooks/data-relation-changed (+1/-0)
hooks/data-relation-departed (+1/-0)
hooks/db-admin-relation-changed (+1/-0)
hooks/db-admin-relation-departed (+1/-0)
hooks/db-relation-changed (+1/-0)
hooks/db-relation-departed (+1/-0)
hooks/defaulthook.py (+65/-0)
hooks/install (+1/-0)
hooks/leader-elected (+1/-0)
hooks/leader-settings-changed (+1/-0)
hooks/local-monitors-relation-changed (+1/-0)
hooks/local-monitors-relation-departed (+1/-0)
hooks/master-relation-changed (+1/-0)
hooks/master-relation-departed (+1/-0)
hooks/nrpe-external-master-relation-changed (+1/-0)
hooks/nrpe-external-master-relation-departed (+1/-0)
hooks/relations/__init__.py (+0/-0)
hooks/relations/syslog/__init__.py (+0/-0)
hooks/relations/syslog/provides.py (+73/-0)
hooks/replication-relation-changed (+1/-0)
hooks/replication-relation-departed (+1/-0)
hooks/start (+1/-0)
hooks/stop (+1/-0)
hooks/syslog-relation-changed (+1/-0)
hooks/syslog-relation-departed (+1/-0)
hooks/upgrade-charm (+1/-0)
lib/charmhelpers/__init__.py (+38/-0)
lib/charmhelpers/cli/__init__.py (+191/-0)
lib/charmhelpers/cli/benchmark.py (+36/-0)
lib/charmhelpers/cli/commands.py (+32/-0)
lib/charmhelpers/cli/hookenv.py (+23/-0)
lib/charmhelpers/cli/host.py (+31/-0)
lib/charmhelpers/cli/unitdata.py (+39/-0)
lib/charmhelpers/context.py (+206/-0)
lib/charmhelpers/contrib/__init__.py (+15/-0)
lib/charmhelpers/contrib/charmsupport/__init__.py (+15/-0)
lib/charmhelpers/contrib/charmsupport/nrpe.py (+398/-0)
lib/charmhelpers/coordinator.py (+607/-0)
lib/charmhelpers/core/__init__.py (+15/-0)
lib/charmhelpers/core/decorators.py (+57/-0)
lib/charmhelpers/core/files.py (+45/-0)
lib/charmhelpers/core/fstab.py (+134/-0)
lib/charmhelpers/core/hookenv.py (+978/-0)
lib/charmhelpers/core/host.py (+641/-0)
lib/charmhelpers/core/hugepage.py (+71/-0)
lib/charmhelpers/core/kernel.py (+68/-0)
lib/charmhelpers/core/services/__init__.py (+18/-0)
lib/charmhelpers/core/services/base.py (+353/-0)
lib/charmhelpers/core/services/helpers.py (+292/-0)
lib/charmhelpers/core/strutils.py (+72/-0)
lib/charmhelpers/core/sysctl.py (+56/-0)
lib/charmhelpers/core/templating.py (+81/-0)
lib/charmhelpers/core/unitdata.py (+521/-0)
lib/charmhelpers/fetch/__init__.py (+464/-0)
lib/charmhelpers/fetch/archiveurl.py (+167/-0)
lib/charmhelpers/fetch/bzrurl.py (+68/-0)
lib/charmhelpers/fetch/giturl.py (+71/-0)
lib/charmhelpers/payload/__init__.py (+17/-0)
lib/charmhelpers/payload/archive.py (+73/-0)
lib/charmhelpers/payload/execd.py (+66/-0)
lib/everyhook.py (+44/-0)
lib/pg_settings_9.5.json (+3145/-0)
lib/pgclient/hooks/hooks.py (+5/-6)
lib/preflight.py (+39/-0)
lib/pypi/charms.reactive-0.3.7.dist-info/DESCRIPTION.rst (+24/-0)
lib/pypi/charms.reactive-0.3.7.dist-info/METADATA (+37/-0)
lib/pypi/charms.reactive-0.3.7.dist-info/RECORD (+15/-0)
lib/pypi/charms.reactive-0.3.7.dist-info/WHEEL (+5/-0)
lib/pypi/charms.reactive-0.3.7.dist-info/metadata.json (+1/-0)
lib/pypi/charms.reactive-0.3.7.dist-info/top_level.txt (+1/-0)
lib/pypi/charms/__init__.py (+2/-0)
lib/pypi/charms/reactive/__init__.py (+72/-0)
lib/pypi/charms/reactive/bus.py (+450/-0)
lib/pypi/charms/reactive/cli.py (+156/-0)
lib/pypi/charms/reactive/decorators.py (+180/-0)
lib/pypi/charms/reactive/helpers.py (+201/-0)
lib/pypi/charms/reactive/relations.py (+664/-0)
reactive/__init__.py (+0/-0)
reactive/apt.py (+176/-0)
reactive/coordinator.py (+70/-0)
reactive/leadership.py (+84/-0)
reactive/postgresql/__init__.py (+0/-0)
reactive/postgresql/client.py (+234/-0)
reactive/postgresql/helpers.py (+120/-0)
reactive/postgresql/metrics.py (+74/-0)
reactive/postgresql/nagios.py (+104/-0)
reactive/postgresql/postgresql.py (+687/-0)
reactive/postgresql/preflight.py (+112/-0)
reactive/postgresql/replication.py (+460/-0)
reactive/postgresql/service.py (+865/-0)
reactive/postgresql/storage.py (+104/-0)
reactive/postgresql/syslog.py (+38/-0)
reactive/postgresql/upgrade.py (+118/-0)
reactive/postgresql/wal_e.py (+152/-0)
reactive/workloadstatus.py (+63/-0)
templates/rsyslog_forward.conf (+3/-3)
tests/test_integration.py (+23/-9)
tests/test_pg_hba_conf.py (+6/-3)
tests/test_postgresql.py (+39/-34)
Reviewer Review Type Date Requested Status
Stuart Bishop Resubmit on 2016-01-14
Review via email: mp+282299@code.launchpad.net

Commit Message

PostgreSQL 9.5 support and port to the charms.reactive framework.

Description of the Change

Port the PostgreSQL charm to the reactive framework.

This is purely the reactive framework changes.

Using the charm builder and splitting things into layers will be done on a future branch.

Making relations more idiomatic (using RelationBase) is also left for a future branch. The existing code did not fit the new model well, so I elected to keep it more old school and reuse it. The Syslog relation alone got redone.

Also adds PostgreSQL 9.5 support since that has been released and will likely be in Xenial. Purely the reactive framework changes and PG 9.5 support.

Also, as you may have noticed, this branch is in git. I'm just a trouble maker.

To post a comment you must log in.
b50e352... by Stuart Bishop on 2016-01-14

Remove obsolete files

Stuart Bishop (stub) wrote :

I'm superseding this with a charm generated from my layered work. This git only MP isn't going to show up in the review queue.

https://code.launchpad.net/~stub/charms/trusty/postgresql/built/+merge/282588

review: Resubmit

Unmerged commits

b50e352... by Stuart Bishop on 2016-01-14

Remove obsolete files

ff42e8e... by Stuart Bishop on 2016-01-08

Block Storage Broker support and charm upgrade fixes

d789f7d... by Stuart Bishop on 2016-01-12

Move storage.py into place

8142b23... by Stuart Bishop on 2016-01-08

PostgreSQL 9.5 support

84ada8e... by Stuart Bishop on 2016-01-04

Work in progress

ac0adb8... by Stuart Bishop on 2016-01-06

sync dependencies

ff53dd9... by Stuart Bishop on 2015-12-12

Syslog relation

501070a... by Stuart Bishop on 2015-12-11

Resync charmhelpers and charms.reactive dependencies

98b9f0e... by Stuart Bishop on 2015-12-11

preflight decorator

1f89fa0... by Stuart Bishop on 2015-12-11

Common validation

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/.gitignore b/.gitignore
2new file mode 100644
3index 0000000..ebfa3ae
4--- /dev/null
5+++ b/.gitignore
6@@ -0,0 +1,5 @@
7+*~
8+*.pyc
9+.coverage
10+bzrsync2.sh
11+lib/pgclient/hooks/charmhelpers
12diff --git a/Makefile b/Makefile
13index c920e64..c325636 100644
14--- a/Makefile
15+++ b/Makefile
16@@ -50,8 +50,8 @@ lint:
17 @echo "Lint check (flake8)"
18 @flake8 -v \
19 --ignore=E402 \
20- --exclude=hooks/charmhelpers,__pycache__ \
21- hooks actions testing tests
22+ --exclude=lib/charmhelpers,lib/pgclient/hooks/charmhelpers,lib/pypi,__pycache__ \
23+ hooks actions testing tests reactive lib
24
25 _co=,
26 _empty=
27@@ -81,17 +81,45 @@ coverage:
28
29 integration:
30 ${TIMING_NOSE} tests/test_integration.py 2>&1 | ts
31+
32+# More overheads, but better progress reporting
33+integration_breakup:
34+ ${NOSE} tests/test_integration.py:PG93Tests 2>&1 | ts
35+ ${NOSE} tests/test_integration.py:PG93MultiTests 2>&1 | ts
36+ ${NOSE} tests/test_integration.py:UpgradedCharmTests 2>&1 | ts
37+ ${NOSE} tests/test_integration.py:PG91Tests 2>&1 | ts
38+ ${NOSE} tests/test_integration.py:PG91MultiTests 2>&1 | ts
39+ ${NOSE} tests/test_integration.py:PG95Tests 2>&1 | ts
40+ ${NOSE} tests/test_integration.py:PG95MultiTests 2>&1 | ts
41+ ${NOSE} tests/test_integration.py:PG94Tests 2>&1 | ts
42+ ${NOSE} tests/test_integration.py:PG94MultiTests 2>&1 | ts
43+ ${NOSE} tests/test_integration.py:PG92Tests 2>&1 | ts
44+ ${NOSE} tests/test_integration.py:PG92MultiTests 2>&1 | ts
45 @echo OK: Integration tests pass `date`
46
47-sync:
48+sync: sync-charmhelpers sync-pypi
49+
50+# Embed from a branch, as we often will need patches applied.
51+sync-charmhelpers:
52 @bzr cat \
53 lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py \
54 > .charm_helpers_sync.py
55+ rm -rf lib/charmhelpers
56 @python .charm_helpers_sync.py -c charm-helpers.yaml
57 @rm .charm_helpers_sync.py
58+ git add -A lib/charmhelpers
59+
60+
61+# Embed pure python pypi dependencies.
62+sync-pypi:
63+ rm -rf lib/pypi
64+ mkdir lib/pypi
65+ pip3 install --no-compile --no-deps -t lib/pypi charms.reactive
66+ git add -A lib/pypi
67
68
69 # These targets are to separate the test output in the Charm CI system
70+# eg. 'make test_integration.py:PG93Tests'
71 test_integration.py%:
72 ${TIMING_NOSE} tests/$@ 2>&1 | ts
73 @echo OK: $@ tests pass `date`
74diff --git a/README.md b/README.md
75index cb4cfe0..92caed1 100644
76--- a/README.md
77+++ b/README.md
78@@ -308,7 +308,7 @@ and set the service configuration settings similar to the following::
79
80 - [PostgreSQL website](http://www.postgresql.org/)
81 - [PostgreSQL bug submission
82- guidelines](http://www.postgresql.org/docs/9.2/static/bug-reporting.html)
83+ guidelines](http://www.postgresql.org/docs/9.5/static/bug-reporting.html)
84 - [PostgreSQL Mailing List](http://www.postgresql.org/list/)
85
86 [1]: https://bugs.launchpad.net/charms/+source/postgresql/+bug/1258485
87diff --git a/charm-helpers.yaml b/charm-helpers.yaml
88index 5e3fbdb..4b08f01 100644
89--- a/charm-helpers.yaml
90+++ b/charm-helpers.yaml
91@@ -1,6 +1,8 @@
92-destination: hooks/charmhelpers
93-branch: lp:~stub/charm-helpers/integration
94+destination: lib/charmhelpers
95+branch: lp:charm-helpers
96+# branch: lp:~stub/charm-helpers/integration
97 include:
98+ - cli
99 - context
100 - coordinator
101 - core
102diff --git a/config.yaml b/config.yaml
103index ddade8d..3d9de2b 100644
104--- a/config.yaml
105+++ b/config.yaml
106@@ -38,18 +38,13 @@ options:
107 type: string
108 description: >
109 Space separated list of extra packages to install.
110- dumpfile_location:
111- default: "None"
112- type: string
113- description: >
114- Path to a dumpfile to load into DB when service is initiated.
115 version:
116 default: ""
117 type: string
118 description: >
119 Version of PostgreSQL that we want to install. Supported versions
120- are "9.1", "9.2", "9.3" & "9.4". The default version for the
121- deployed Ubuntu release is used when the version is not specified.
122+ are "9.1", "9.2", "9.3", "9.4" & "9.5". The default version for the
123+ deployed Ubuntu release is used when the version is unspecified.
124 extra_pg_conf:
125 # The defaults here match the defaults chosen by the charm,
126 # so removing them will not change them. They are listed
127@@ -67,7 +62,7 @@ options:
128 log_disconnections=true
129 log_autovacuum_min_duration=-1
130 log_line_prefix='%t [%p]: [%l-1] db=%d,user=%u '
131- archive_mode=true
132+ archive_mode=on
133 archive_command='/bin/true'
134 hot_standby=true
135 max_wal_senders=80
136@@ -129,6 +124,11 @@ options:
137 default: 7
138 type: int
139 description: Number of daily backups to retain.
140+ backup_dir:
141+ default: "/var/lib/postgresql/backups"
142+ type: string
143+ description: >
144+ Directory to place backups in.
145 nagios_context:
146 default: "juju"
147 type: string
148@@ -609,10 +609,4 @@ options:
149 type: float
150 description: >
151 DEPRECATED. Use extra_pg_conf. Random page cost
152- backup_dir:
153- default: "/var/lib/postgresql/backups"
154- type: string
155- description: >
156- DEPRECATED. Directory to place backups in. If you change this,
157- your backups will go to this path and not the 'backups' Juju
158- storage mount.
159+
160diff --git a/hooks/bootstrap.py b/hooks/bootstrap.py
161deleted file mode 100644
162index 5fe4ebd..0000000
163--- a/hooks/bootstrap.py
164+++ /dev/null
165@@ -1,57 +0,0 @@
166-#!/usr/bin/python3
167-
168-# Copyright 2015 Canonical Ltd.
169-#
170-# This file is part of the PostgreSQL Charm for Juju.
171-#
172-# This program is free software: you can redistribute it and/or modify
173-# it under the terms of the GNU General Public License version 3, as
174-# published by the Free Software Foundation.
175-#
176-# This program is distributed in the hope that it will be useful, but
177-# WITHOUT ANY WARRANTY; without even the implied warranties of
178-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
179-# PURPOSE. See the GNU General Public License for more details.
180-#
181-# You should have received a copy of the GNU General Public License
182-# along with this program. If not, see <http://www.gnu.org/licenses/>.
183-
184-from charmhelpers import fetch
185-from charmhelpers.core import hookenv
186-
187-
188-def bootstrap():
189- try:
190- import psycopg2 # NOQA: flake8
191- import jinja2 # NOQA: flake8
192- except ImportError:
193- packages = ['python3-psycopg2', 'python3-jinja2']
194- fetch.apt_install(packages, fatal=True)
195- import psycopg2 # NOQA: flake8
196-
197-
198-def block_on_bad_juju():
199- if not hookenv.has_juju_version('1.24'):
200- hookenv.status_set('blocked', 'Requires Juju 1.24 or higher')
201- # Error state, since we don't have 1.24 to give a nice blocked state.
202- raise SystemExit(1)
203-
204-
205-def upgrade_charm():
206- block_on_bad_juju()
207- # This needs to be imported after bootstrap() or required Python
208- # packages may not have been installed.
209- import upgrade
210- upgrade.upgrade_charm()
211-
212-
213-def default_hook():
214- block_on_bad_juju()
215- # This needs to be imported after bootstrap() or required Python
216- # packages may not have been installed.
217- import definitions
218-
219- hookenv.log('*** Start {!r} hook'.format(hookenv.hook_name()))
220- sm = definitions.get_service_manager()
221- sm.manage()
222- hookenv.log('*** End {!r} hook'.format(hookenv.hook_name()))
223diff --git a/hooks/charmhelpers/__init__.py b/hooks/charmhelpers/__init__.py
224deleted file mode 100644
225index f72e7f8..0000000
226--- a/hooks/charmhelpers/__init__.py
227+++ /dev/null
228@@ -1,38 +0,0 @@
229-# Copyright 2014-2015 Canonical Limited.
230-#
231-# This file is part of charm-helpers.
232-#
233-# charm-helpers is free software: you can redistribute it and/or modify
234-# it under the terms of the GNU Lesser General Public License version 3 as
235-# published by the Free Software Foundation.
236-#
237-# charm-helpers is distributed in the hope that it will be useful,
238-# but WITHOUT ANY WARRANTY; without even the implied warranty of
239-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
240-# GNU Lesser General Public License for more details.
241-#
242-# You should have received a copy of the GNU Lesser General Public License
243-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
244-
245-# Bootstrap charm-helpers, installing its dependencies if necessary using
246-# only standard libraries.
247-import subprocess
248-import sys
249-
250-try:
251- import six # flake8: noqa
252-except ImportError:
253- if sys.version_info.major == 2:
254- subprocess.check_call(['apt-get', 'install', '-y', 'python-six'])
255- else:
256- subprocess.check_call(['apt-get', 'install', '-y', 'python3-six'])
257- import six # flake8: noqa
258-
259-try:
260- import yaml # flake8: noqa
261-except ImportError:
262- if sys.version_info.major == 2:
263- subprocess.check_call(['apt-get', 'install', '-y', 'python-yaml'])
264- else:
265- subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
266- import yaml # flake8: noqa
267diff --git a/hooks/charmhelpers/context.py b/hooks/charmhelpers/context.py
268deleted file mode 100644
269index 7035d04..0000000
270--- a/hooks/charmhelpers/context.py
271+++ /dev/null
272@@ -1,206 +0,0 @@
273-# Copyright 2015 Canonical Limited.
274-#
275-# This file is part of charm-helpers.
276-#
277-# charm-helpers is free software: you can redistribute it and/or modify
278-# it under the terms of the GNU Lesser General Public License version 3 as
279-# published by the Free Software Foundation.
280-#
281-# charm-helpers is distributed in the hope that it will be useful,
282-# but WITHOUT ANY WARRANTY; without even the implied warranty of
283-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
284-# GNU Lesser General Public License for more details.
285-#
286-# You should have received a copy of the GNU Lesser General Public License
287-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
288-'''
289-A Pythonic API to interact with the charm hook environment.
290-
291-:author: Stuart Bishop <stuart.bishop@canonical.com>
292-'''
293-
294-import six
295-
296-from charmhelpers.core import hookenv
297-
298-from collections import OrderedDict
299-if six.PY3:
300- from collections import UserDict # pragma: nocover
301-else:
302- from UserDict import IterableUserDict as UserDict # pragma: nocover
303-
304-
305-class Relations(OrderedDict):
306- '''Mapping relation name -> relation id -> Relation.
307-
308- >>> rels = Relations()
309- >>> rels['sprog']['sprog:12']['client/6']['widget']
310- 'remote widget'
311- >>> rels['sprog']['sprog:12'].local['widget'] = 'local widget'
312- >>> rels['sprog']['sprog:12'].local['widget']
313- 'local widget'
314- >>> rels.peer.local['widget']
315- 'local widget on the peer relation'
316- '''
317- def __init__(self):
318- super(Relations, self).__init__()
319- for relname in sorted(hookenv.relation_types()):
320- self[relname] = OrderedDict()
321- relids = hookenv.relation_ids(relname)
322- relids.sort(key=lambda x: int(x.split(':', 1)[-1]))
323- for relid in relids:
324- self[relname][relid] = Relation(relid)
325-
326- @property
327- def peer(self):
328- peer_relid = hookenv.peer_relation_id()
329- for rels in self.values():
330- if peer_relid in rels:
331- return rels[peer_relid]
332-
333-
334-class Relation(OrderedDict):
335- '''Mapping of unit -> remote RelationInfo for a relation.
336-
337- This is an OrderedDict mapping, ordered numerically by
338- by unit number.
339-
340- Also provides access to the local RelationInfo, and peer RelationInfo
341- instances by the 'local' and 'peers' attributes.
342-
343- >>> r = Relation('sprog:12')
344- >>> r.keys()
345- ['client/9', 'client/10'] # Ordered numerically
346- >>> r['client/10']['widget'] # A remote RelationInfo setting
347- 'remote widget'
348- >>> r.local['widget'] # The local RelationInfo setting
349- 'local widget'
350- '''
351- relid = None # The relation id.
352- relname = None # The relation name (also known as relation type).
353- service = None # The remote service name, if known.
354- local = None # The local end's RelationInfo.
355- peers = None # Map of peer -> RelationInfo. None if no peer relation.
356-
357- def __init__(self, relid):
358- remote_units = hookenv.related_units(relid)
359- remote_units.sort(key=lambda u: int(u.split('/', 1)[-1]))
360- super(Relation, self).__init__((unit, RelationInfo(relid, unit))
361- for unit in remote_units)
362-
363- self.relname = relid.split(':', 1)[0]
364- self.relid = relid
365- self.local = RelationInfo(relid, hookenv.local_unit())
366-
367- for relinfo in self.values():
368- self.service = relinfo.service
369- break
370-
371- # If we have peers, and they have joined both the provided peer
372- # relation and this relation, we can peek at their data too.
373- # This is useful for creating consensus without leadership.
374- peer_relid = hookenv.peer_relation_id()
375- if peer_relid and peer_relid != relid:
376- peers = hookenv.related_units(peer_relid)
377- if peers:
378- peers.sort(key=lambda u: int(u.split('/', 1)[-1]))
379- self.peers = OrderedDict((peer, RelationInfo(relid, peer))
380- for peer in peers)
381- else:
382- self.peers = OrderedDict()
383- else:
384- self.peers = None
385-
386- def __str__(self):
387- return '{} ({})'.format(self.relid, self.service)
388-
389-
390-class RelationInfo(UserDict):
391- '''The bag of data at an end of a relation.
392-
393- Every unit participating in a relation has a single bag of
394- data associated with that relation. This is that bag.
395-
396- The bag of data for the local unit may be updated. Remote data
397- is immutable and will remain static for the duration of the hook.
398-
399- Changes made to the local units relation data only become visible
400- to other units after the hook completes successfully. If the hook
401- does not complete successfully, the changes are rolled back.
402-
403- Unlike standard Python mappings, setting an item to None is the
404- same as deleting it.
405-
406- >>> relinfo = RelationInfo('db:12') # Default is the local unit.
407- >>> relinfo['user'] = 'fred'
408- >>> relinfo['user']
409- 'fred'
410- >>> relinfo['user'] = None
411- >>> 'fred' in relinfo
412- False
413-
414- This class wraps hookenv.relation_get and hookenv.relation_set.
415- All caching is left up to these two methods to avoid synchronization
416- issues. Data is only loaded on demand.
417- '''
418- relid = None # The relation id.
419- relname = None # The relation name (also know as the relation type).
420- unit = None # The unit id.
421- number = None # The unit number (integer).
422- service = None # The service name.
423-
424- def __init__(self, relid, unit):
425- self.relname = relid.split(':', 1)[0]
426- self.relid = relid
427- self.unit = unit
428- self.service, num = self.unit.split('/', 1)
429- self.number = int(num)
430-
431- def __str__(self):
432- return '{} ({})'.format(self.relid, self.unit)
433-
434- @property
435- def data(self):
436- return hookenv.relation_get(rid=self.relid, unit=self.unit)
437-
438- def __setitem__(self, key, value):
439- if self.unit != hookenv.local_unit():
440- raise TypeError('Attempting to set {} on remote unit {}'
441- ''.format(key, self.unit))
442- if value is not None and not isinstance(value, six.string_types):
443- # We don't do implicit casting. This would cause simple
444- # types like integers to be read back as strings in subsequent
445- # hooks, and mutable types would require a lot of wrapping
446- # to ensure relation-set gets called when they are mutated.
447- raise ValueError('Only string values allowed')
448- hookenv.relation_set(self.relid, {key: value})
449-
450- def __delitem__(self, key):
451- # Deleting a key and setting it to null is the same thing in
452- # Juju relations.
453- self[key] = None
454-
455-
456-class Leader(UserDict):
457- def __init__(self):
458- pass # Don't call superclass initializer, as it will nuke self.data
459-
460- @property
461- def data(self):
462- return hookenv.leader_get()
463-
464- def __setitem__(self, key, value):
465- if not hookenv.is_leader():
466- raise TypeError('Not the leader. Cannot change leader settings.')
467- if value is not None and not isinstance(value, six.string_types):
468- # We don't do implicit casting. This would cause simple
469- # types like integers to be read back as strings in subsequent
470- # hooks, and mutable types would require a lot of wrapping
471- # to ensure leader-set gets called when they are mutated.
472- raise ValueError('Only string values allowed')
473- hookenv.leader_set({key: value})
474-
475- def __delitem__(self, key):
476- # Deleting a key and setting it to null is the same thing in
477- # Juju leadership settings.
478- self[key] = None
479diff --git a/hooks/charmhelpers/contrib/__init__.py b/hooks/charmhelpers/contrib/__init__.py
480deleted file mode 100644
481index d1400a0..0000000
482--- a/hooks/charmhelpers/contrib/__init__.py
483+++ /dev/null
484@@ -1,15 +0,0 @@
485-# Copyright 2014-2015 Canonical Limited.
486-#
487-# This file is part of charm-helpers.
488-#
489-# charm-helpers is free software: you can redistribute it and/or modify
490-# it under the terms of the GNU Lesser General Public License version 3 as
491-# published by the Free Software Foundation.
492-#
493-# charm-helpers is distributed in the hope that it will be useful,
494-# but WITHOUT ANY WARRANTY; without even the implied warranty of
495-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
496-# GNU Lesser General Public License for more details.
497-#
498-# You should have received a copy of the GNU Lesser General Public License
499-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
500diff --git a/hooks/charmhelpers/contrib/charmsupport/__init__.py b/hooks/charmhelpers/contrib/charmsupport/__init__.py
501deleted file mode 100644
502index d1400a0..0000000
503--- a/hooks/charmhelpers/contrib/charmsupport/__init__.py
504+++ /dev/null
505@@ -1,15 +0,0 @@
506-# Copyright 2014-2015 Canonical Limited.
507-#
508-# This file is part of charm-helpers.
509-#
510-# charm-helpers is free software: you can redistribute it and/or modify
511-# it under the terms of the GNU Lesser General Public License version 3 as
512-# published by the Free Software Foundation.
513-#
514-# charm-helpers is distributed in the hope that it will be useful,
515-# but WITHOUT ANY WARRANTY; without even the implied warranty of
516-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
517-# GNU Lesser General Public License for more details.
518-#
519-# You should have received a copy of the GNU Lesser General Public License
520-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
521diff --git a/hooks/charmhelpers/contrib/charmsupport/nrpe.py b/hooks/charmhelpers/contrib/charmsupport/nrpe.py
522deleted file mode 100644
523index 95a79c2..0000000
524--- a/hooks/charmhelpers/contrib/charmsupport/nrpe.py
525+++ /dev/null
526@@ -1,360 +0,0 @@
527-# Copyright 2014-2015 Canonical Limited.
528-#
529-# This file is part of charm-helpers.
530-#
531-# charm-helpers is free software: you can redistribute it and/or modify
532-# it under the terms of the GNU Lesser General Public License version 3 as
533-# published by the Free Software Foundation.
534-#
535-# charm-helpers is distributed in the hope that it will be useful,
536-# but WITHOUT ANY WARRANTY; without even the implied warranty of
537-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
538-# GNU Lesser General Public License for more details.
539-#
540-# You should have received a copy of the GNU Lesser General Public License
541-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
542-
543-"""Compatibility with the nrpe-external-master charm"""
544-# Copyright 2012 Canonical Ltd.
545-#
546-# Authors:
547-# Matthew Wedgwood <matthew.wedgwood@canonical.com>
548-
549-import subprocess
550-import pwd
551-import grp
552-import os
553-import glob
554-import shutil
555-import re
556-import shlex
557-import yaml
558-
559-from charmhelpers.core.hookenv import (
560- config,
561- local_unit,
562- log,
563- relation_ids,
564- relation_set,
565- relations_of_type,
566-)
567-
568-from charmhelpers.core.host import service
569-
570-# This module adds compatibility with the nrpe-external-master and plain nrpe
571-# subordinate charms. To use it in your charm:
572-#
573-# 1. Update metadata.yaml
574-#
575-# provides:
576-# (...)
577-# nrpe-external-master:
578-# interface: nrpe-external-master
579-# scope: container
580-#
581-# and/or
582-#
583-# provides:
584-# (...)
585-# local-monitors:
586-# interface: local-monitors
587-# scope: container
588-
589-#
590-# 2. Add the following to config.yaml
591-#
592-# nagios_context:
593-# default: "juju"
594-# type: string
595-# description: |
596-# Used by the nrpe subordinate charms.
597-# A string that will be prepended to instance name to set the host name
598-# in nagios. So for instance the hostname would be something like:
599-# juju-myservice-0
600-# If you're running multiple environments with the same services in them
601-# this allows you to differentiate between them.
602-# nagios_servicegroups:
603-# default: ""
604-# type: string
605-# description: |
606-# A comma-separated list of nagios servicegroups.
607-# If left empty, the nagios_context will be used as the servicegroup
608-#
609-# 3. Add custom checks (Nagios plugins) to files/nrpe-external-master
610-#
611-# 4. Update your hooks.py with something like this:
612-#
613-# from charmsupport.nrpe import NRPE
614-# (...)
615-# def update_nrpe_config():
616-# nrpe_compat = NRPE()
617-# nrpe_compat.add_check(
618-# shortname = "myservice",
619-# description = "Check MyService",
620-# check_cmd = "check_http -w 2 -c 10 http://localhost"
621-# )
622-# nrpe_compat.add_check(
623-# "myservice_other",
624-# "Check for widget failures",
625-# check_cmd = "/srv/myapp/scripts/widget_check"
626-# )
627-# nrpe_compat.write()
628-#
629-# def config_changed():
630-# (...)
631-# update_nrpe_config()
632-#
633-# def nrpe_external_master_relation_changed():
634-# update_nrpe_config()
635-#
636-# def local_monitors_relation_changed():
637-# update_nrpe_config()
638-#
639-# 5. ln -s hooks.py nrpe-external-master-relation-changed
640-# ln -s hooks.py local-monitors-relation-changed
641-
642-
643-class CheckException(Exception):
644- pass
645-
646-
647-class Check(object):
648- shortname_re = '[A-Za-z0-9-_]+$'
649- service_template = ("""
650-#---------------------------------------------------
651-# This file is Juju managed
652-#---------------------------------------------------
653-define service {{
654- use active-service
655- host_name {nagios_hostname}
656- service_description {nagios_hostname}[{shortname}] """
657- """{description}
658- check_command check_nrpe!{command}
659- servicegroups {nagios_servicegroup}
660-}}
661-""")
662-
663- def __init__(self, shortname, description, check_cmd):
664- super(Check, self).__init__()
665- # XXX: could be better to calculate this from the service name
666- if not re.match(self.shortname_re, shortname):
667- raise CheckException("shortname must match {}".format(
668- Check.shortname_re))
669- self.shortname = shortname
670- self.command = "check_{}".format(shortname)
671- # Note: a set of invalid characters is defined by the
672- # Nagios server config
673- # The default is: illegal_object_name_chars=`~!$%^&*"|'<>?,()=
674- self.description = description
675- self.check_cmd = self._locate_cmd(check_cmd)
676-
677- def _locate_cmd(self, check_cmd):
678- search_path = (
679- '/usr/lib/nagios/plugins',
680- '/usr/local/lib/nagios/plugins',
681- )
682- parts = shlex.split(check_cmd)
683- for path in search_path:
684- if os.path.exists(os.path.join(path, parts[0])):
685- command = os.path.join(path, parts[0])
686- if len(parts) > 1:
687- command += " " + " ".join(parts[1:])
688- return command
689- log('Check command not found: {}'.format(parts[0]))
690- return ''
691-
692- def write(self, nagios_context, hostname, nagios_servicegroups):
693- nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
694- self.command)
695- with open(nrpe_check_file, 'w') as nrpe_check_config:
696- nrpe_check_config.write("# check {}\n".format(self.shortname))
697- nrpe_check_config.write("command[{}]={}\n".format(
698- self.command, self.check_cmd))
699-
700- if not os.path.exists(NRPE.nagios_exportdir):
701- log('Not writing service config as {} is not accessible'.format(
702- NRPE.nagios_exportdir))
703- else:
704- self.write_service_config(nagios_context, hostname,
705- nagios_servicegroups)
706-
707- def write_service_config(self, nagios_context, hostname,
708- nagios_servicegroups):
709- for f in os.listdir(NRPE.nagios_exportdir):
710- if re.search('.*{}.cfg'.format(self.command), f):
711- os.remove(os.path.join(NRPE.nagios_exportdir, f))
712-
713- templ_vars = {
714- 'nagios_hostname': hostname,
715- 'nagios_servicegroup': nagios_servicegroups,
716- 'description': self.description,
717- 'shortname': self.shortname,
718- 'command': self.command,
719- }
720- nrpe_service_text = Check.service_template.format(**templ_vars)
721- nrpe_service_file = '{}/service__{}_{}.cfg'.format(
722- NRPE.nagios_exportdir, hostname, self.command)
723- with open(nrpe_service_file, 'w') as nrpe_service_config:
724- nrpe_service_config.write(str(nrpe_service_text))
725-
726- def run(self):
727- subprocess.call(self.check_cmd)
728-
729-
730-class NRPE(object):
731- nagios_logdir = '/var/log/nagios'
732- nagios_exportdir = '/var/lib/nagios/export'
733- nrpe_confdir = '/etc/nagios/nrpe.d'
734-
735- def __init__(self, hostname=None):
736- super(NRPE, self).__init__()
737- self.config = config()
738- self.nagios_context = self.config['nagios_context']
739- if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']:
740- self.nagios_servicegroups = self.config['nagios_servicegroups']
741- else:
742- self.nagios_servicegroups = self.nagios_context
743- self.unit_name = local_unit().replace('/', '-')
744- if hostname:
745- self.hostname = hostname
746- else:
747- self.hostname = "{}-{}".format(self.nagios_context, self.unit_name)
748- self.checks = []
749-
750- def add_check(self, *args, **kwargs):
751- self.checks.append(Check(*args, **kwargs))
752-
753- def write(self):
754- try:
755- nagios_uid = pwd.getpwnam('nagios').pw_uid
756- nagios_gid = grp.getgrnam('nagios').gr_gid
757- except:
758- log("Nagios user not set up, nrpe checks not updated")
759- return
760-
761- if not os.path.exists(NRPE.nagios_logdir):
762- os.mkdir(NRPE.nagios_logdir)
763- os.chown(NRPE.nagios_logdir, nagios_uid, nagios_gid)
764-
765- nrpe_monitors = {}
766- monitors = {"monitors": {"remote": {"nrpe": nrpe_monitors}}}
767- for nrpecheck in self.checks:
768- nrpecheck.write(self.nagios_context, self.hostname,
769- self.nagios_servicegroups)
770- nrpe_monitors[nrpecheck.shortname] = {
771- "command": nrpecheck.command,
772- }
773-
774- service('restart', 'nagios-nrpe-server')
775-
776- monitor_ids = relation_ids("local-monitors") + \
777- relation_ids("nrpe-external-master")
778- for rid in monitor_ids:
779- relation_set(relation_id=rid, monitors=yaml.dump(monitors))
780-
781-
782-def get_nagios_hostcontext(relation_name='nrpe-external-master'):
783- """
784- Query relation with nrpe subordinate, return the nagios_host_context
785-
786- :param str relation_name: Name of relation nrpe sub joined to
787- """
788- for rel in relations_of_type(relation_name):
789- if 'nagios_hostname' in rel:
790- return rel['nagios_host_context']
791-
792-
793-def get_nagios_hostname(relation_name='nrpe-external-master'):
794- """
795- Query relation with nrpe subordinate, return the nagios_hostname
796-
797- :param str relation_name: Name of relation nrpe sub joined to
798- """
799- for rel in relations_of_type(relation_name):
800- if 'nagios_hostname' in rel:
801- return rel['nagios_hostname']
802-
803-
804-def get_nagios_unit_name(relation_name='nrpe-external-master'):
805- """
806- Return the nagios unit name prepended with host_context if needed
807-
808- :param str relation_name: Name of relation nrpe sub joined to
809- """
810- host_context = get_nagios_hostcontext(relation_name)
811- if host_context:
812- unit = "%s:%s" % (host_context, local_unit())
813- else:
814- unit = local_unit()
815- return unit
816-
817-
818-def add_init_service_checks(nrpe, services, unit_name):
819- """
820- Add checks for each service in list
821-
822- :param NRPE nrpe: NRPE object to add check to
823- :param list services: List of services to check
824- :param str unit_name: Unit name to use in check description
825- """
826- for svc in services:
827- upstart_init = '/etc/init/%s.conf' % svc
828- sysv_init = '/etc/init.d/%s' % svc
829- if os.path.exists(upstart_init):
830- nrpe.add_check(
831- shortname=svc,
832- description='process check {%s}' % unit_name,
833- check_cmd='check_upstart_job %s' % svc
834- )
835- elif os.path.exists(sysv_init):
836- cronpath = '/etc/cron.d/nagios-service-check-%s' % svc
837- cron_file = ('*/5 * * * * root '
838- '/usr/local/lib/nagios/plugins/check_exit_status.pl '
839- '-s /etc/init.d/%s status > '
840- '/var/lib/nagios/service-check-%s.txt\n' % (svc,
841- svc)
842- )
843- f = open(cronpath, 'w')
844- f.write(cron_file)
845- f.close()
846- nrpe.add_check(
847- shortname=svc,
848- description='process check {%s}' % unit_name,
849- check_cmd='check_status_file.py -f '
850- '/var/lib/nagios/service-check-%s.txt' % svc,
851- )
852-
853-
854-def copy_nrpe_checks():
855- """
856- Copy the nrpe checks into place
857-
858- """
859- NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins'
860- nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks',
861- 'charmhelpers', 'contrib', 'openstack',
862- 'files')
863-
864- if not os.path.exists(NAGIOS_PLUGINS):
865- os.makedirs(NAGIOS_PLUGINS)
866- for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")):
867- if os.path.isfile(fname):
868- shutil.copy2(fname,
869- os.path.join(NAGIOS_PLUGINS, os.path.basename(fname)))
870-
871-
872-def add_haproxy_checks(nrpe, unit_name):
873- """
874- Add checks for each service in list
875-
876- :param NRPE nrpe: NRPE object to add check to
877- :param str unit_name: Unit name to use in check description
878- """
879- nrpe.add_check(
880- shortname='haproxy_servers',
881- description='Check HAProxy {%s}' % unit_name,
882- check_cmd='check_haproxy.sh')
883- nrpe.add_check(
884- shortname='haproxy_queue',
885- description='Check HAProxy queue depth {%s}' % unit_name,
886- check_cmd='check_haproxy_queue_depth.sh')
887diff --git a/hooks/charmhelpers/coordinator.py b/hooks/charmhelpers/coordinator.py
888deleted file mode 100644
889index 0fad251..0000000
890--- a/hooks/charmhelpers/coordinator.py
891+++ /dev/null
892@@ -1,607 +0,0 @@
893-# Copyright 2014-2015 Canonical Limited.
894-#
895-# This file is part of charm-helpers.
896-#
897-# charm-helpers is free software: you can redistribute it and/or modify
898-# it under the terms of the GNU Lesser General Public License version 3 as
899-# published by the Free Software Foundation.
900-#
901-# charm-helpers is distributed in the hope that it will be useful,
902-# but WITHOUT ANY WARRANTY; without even the implied warranty of
903-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
904-# GNU Lesser General Public License for more details.
905-#
906-# You should have received a copy of the GNU Lesser General Public License
907-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
908-'''
909-The coordinator module allows you to use Juju's leadership feature to
910-coordinate operations between units of a service.
911-
912-Behavior is defined in subclasses of coordinator.BaseCoordinator.
913-One implementation is provided (coordinator.Serial), which allows an
914-operation to be run on a single unit at a time, on a first come, first
915-served basis. You can trivially define more complex behavior by
916-subclassing BaseCoordinator or Serial.
917-
918-:author: Stuart Bishop <stuart.bishop@canonical.com>
919-
920-
921-Services Framework Usage
922-========================
923-
924-Ensure a peer relation is defined in metadata.yaml. Instantiate a
925-BaseCoordinator subclass before invoking ServiceManager.manage().
926-Ensure that ServiceManager.manage() is wired up to the leader-elected,
927-leader-settings-changed, peer relation-changed and peer
928-relation-departed hooks in addition to any other hooks you need, or your
929-service will deadlock.
930-
931-Ensure calls to acquire() are guarded, so that locks are only requested
932-when they are really needed (and thus hooks only triggered when necessary).
933-Failing to do this and calling acquire() unconditionally will put your unit
934-into a hook loop. Calls to granted() do not need to be guarded.
935-
936-For example::
937-
938- from charmhelpers.core import hookenv, services
939- from charmhelpers import coordinator
940-
941- def maybe_restart(servicename):
942- serial = coordinator.Serial()
943- if needs_restart():
944- serial.acquire('restart')
945- if serial.granted('restart'):
946- hookenv.service_restart(servicename)
947-
948- services = [dict(service='servicename',
949- data_ready=[maybe_restart])]
950-
951- if __name__ == '__main__':
952- _ = coordinator.Serial() # Must instantiate before manager.manage()
953- manager = services.ServiceManager(services)
954- manager.manage()
955-
956-
957-You can implement a similar pattern using a decorator. If the lock has
958-not been granted, an attempt to acquire() it will be made if the guard
959-function returns True. If the lock has been granted, the decorated function
960-is run as normal::
961-
962- from charmhelpers.core import hookenv, services
963- from charmhelpers import coordinator
964-
965- serial = coordinator.Serial() # Global, instatiated on module import.
966-
967- def needs_restart():
968- [ ... Introspect state. Return True if restart is needed ... ]
969-
970- @serial.require('restart', needs_restart)
971- def maybe_restart(servicename):
972- hookenv.service_restart(servicename)
973-
974- services = [dict(service='servicename',
975- data_ready=[maybe_restart])]
976-
977- if __name__ == '__main__':
978- manager = services.ServiceManager(services)
979- manager.manage()
980-
981-
982-Traditional Usage
983-=================
984-
985-Ensure a peer relationis defined in metadata.yaml.
986-
987-If you are using charmhelpers.core.hookenv.Hooks, ensure that a
988-BaseCoordinator subclass is instantiated before calling Hooks.execute.
989-
990-If you are not using charmhelpers.core.hookenv.Hooks, ensure
991-that a BaseCoordinator subclass is instantiated and its handle()
992-method called at the start of all your hooks.
993-
994-For example::
995-
996- import sys
997- from charmhelpers.core import hookenv
998- from charmhelpers import coordinator
999-
1000- hooks = hookenv.Hooks()
1001-
1002- def maybe_restart():
1003- serial = coordinator.Serial()
1004- if serial.granted('restart'):
1005- hookenv.service_restart('myservice')
1006-
1007- @hooks.hook
1008- def config_changed():
1009- update_config()
1010- serial = coordinator.Serial()
1011- if needs_restart():
1012- serial.acquire('restart'):
1013- maybe_restart()
1014-
1015- # Cluster hooks must be wired up.
1016- @hooks.hook('cluster-relation-changed', 'cluster-relation-departed')
1017- def cluster_relation_changed():
1018- maybe_restart()
1019-
1020- # Leader hooks must be wired up.
1021- @hooks.hook('leader-elected', 'leader-settings-changed')
1022- def leader_settings_changed():
1023- maybe_restart()
1024-
1025- [ ... repeat for *all* other hooks you are using ... ]
1026-
1027- if __name__ == '__main__':
1028- _ = coordinator.Serial() # Must instantiate before execute()
1029- hooks.execute(sys.argv)
1030-
1031-
1032-You can also use the require decorator. If the lock has not been granted,
1033-an attempt to acquire() it will be made if the guard function returns True.
1034-If the lock has been granted, the decorated function is run as normal::
1035-
1036- from charmhelpers.core import hookenv
1037-
1038- hooks = hookenv.Hooks()
1039- serial = coordinator.Serial() # Must instantiate before execute()
1040-
1041- @require('restart', needs_restart)
1042- def maybe_restart():
1043- hookenv.service_restart('myservice')
1044-
1045- @hooks.hook('install', 'config-changed', 'upgrade-charm',
1046- # Peer and leader hooks must be wired up.
1047- 'cluster-relation-changed', 'cluster-relation-departed',
1048- 'leader-elected', 'leader-settings-changed')
1049- def default_hook():
1050- [...]
1051- maybe_restart()
1052-
1053- if __name__ == '__main__':
1054- hooks.execute()
1055-
1056-
1057-Details
1058-=======
1059-
1060-A simple API is provided similar to traditional locking APIs. A lock
1061-may be requested using the acquire() method, and the granted() method
1062-may be used do to check if a lock previously requested by acquire() has
1063-been granted. It doesn't matter how many times acquire() is called in a
1064-hook.
1065-
1066-Locks are released at the end of the hook they are acquired in. This may
1067-be the current hook if the unit is leader and the lock is free. It is
1068-more likely a future hook (probably leader-settings-changed, possibly
1069-the peer relation-changed or departed hook, potentially any hook).
1070-
1071-Whenever a charm needs to perform a coordinated action it will acquire()
1072-the lock and perform the action immediately if acquisition is
1073-successful. It will also need to perform the same action in every other
1074-hook if the lock has been granted.
1075-
1076-
1077-Grubby Details
1078---------------
1079-
1080-Why do you need to be able to perform the same action in every hook?
1081-If the unit is the leader, then it may be able to grant its own lock
1082-and perform the action immediately in the source hook. If the unit is
1083-the leader and cannot immediately grant the lock, then its only
1084-guaranteed chance of acquiring the lock is in the peer relation-joined,
1085-relation-changed or peer relation-departed hooks when another unit has
1086-released it (the only channel to communicate to the leader is the peer
1087-relation). If the unit is not the leader, then it is unlikely the lock
1088-is granted in the source hook (a previous hook must have also made the
1089-request for this to happen). A non-leader is notified about the lock via
1090-leader settings. These changes may be visible in any hook, even before
1091-the leader-settings-changed hook has been invoked. Or the requesting
1092-unit may be promoted to leader after making a request, in which case the
1093-lock may be granted in leader-elected or in a future peer
1094-relation-changed or relation-departed hook.
1095-
1096-This could be simpler if leader-settings-changed was invoked on the
1097-leader. We could then never grant locks except in
1098-leader-settings-changed hooks giving one place for the operation to be
1099-performed. Unfortunately this is not the case with Juju 1.23 leadership.
1100-
1101-But of course, this doesn't really matter to most people as most people
1102-seem to prefer the Services Framework or similar reset-the-world
1103-approaches, rather than the twisty maze of attempting to deduce what
1104-should be done based on what hook happens to be running (which always
1105-seems to evolve into reset-the-world anyway when the charm grows beyond
1106-the trivial).
1107-
1108-I chose not to implement a callback model, where a callback was passed
1109-to acquire to be executed when the lock is granted, because the callback
1110-may become invalid between making the request and the lock being granted
1111-due to an upgrade-charm being run in the interim. And it would create
1112-restrictions, such no lambdas, callback defined at the top level of a
1113-module, etc. Still, we could implement it on top of what is here, eg.
1114-by adding a defer decorator that stores a pickle of itself to disk and
1115-have BaseCoordinator unpickle and execute them when the locks are granted.
1116-'''
1117-from datetime import datetime
1118-from functools import wraps
1119-import json
1120-import os.path
1121-
1122-from six import with_metaclass
1123-
1124-from charmhelpers.core import hookenv
1125-
1126-
1127-# We make BaseCoordinator and subclasses singletons, so that if we
1128-# need to spill to local storage then only a single instance does so,
1129-# rather than having multiple instances stomp over each other.
1130-class Singleton(type):
1131- _instances = {}
1132-
1133- def __call__(cls, *args, **kwargs):
1134- if cls not in cls._instances:
1135- cls._instances[cls] = super(Singleton, cls).__call__(*args,
1136- **kwargs)
1137- return cls._instances[cls]
1138-
1139-
1140-class BaseCoordinator(with_metaclass(Singleton, object)):
1141- relid = None # Peer relation-id, set by __init__
1142- relname = None
1143-
1144- grants = None # self.grants[unit][lock] == timestamp
1145- requests = None # self.requests[unit][lock] == timestamp
1146-
1147- def __init__(self, relation_key='coordinator', peer_relation_name=None):
1148- '''Instatiate a Coordinator.
1149-
1150- Data is stored on the peer relation and in leadership storage
1151- under the provided relation_key.
1152-
1153- The peer relation is identified by peer_relation_name, and defaults
1154- to the first one found in metadata.yaml.
1155- '''
1156- # Most initialization is deferred, since invoking hook tools from
1157- # the constructor makes testing hard.
1158- self.key = relation_key
1159- self.relname = peer_relation_name
1160- hookenv.atstart(self.initialize)
1161-
1162- # Ensure that handle() is called, without placing that burden on
1163- # the charm author. They still need to do this manually if they
1164- # are not using a hook framework.
1165- hookenv.atstart(self.handle)
1166-
1167- def initialize(self):
1168- if self.requests is not None:
1169- return # Already initialized.
1170-
1171- assert hookenv.has_juju_version('1.23'), 'Needs Juju 1.23+'
1172-
1173- if self.relname is None:
1174- self.relname = _implicit_peer_relation_name()
1175-
1176- relids = hookenv.relation_ids(self.relname)
1177- if relids:
1178- self.relid = sorted(relids)[0]
1179-
1180- # Load our state, from leadership, the peer relationship, and maybe
1181- # local state as a fallback. Populates self.requests and self.grants.
1182- self._load_state()
1183- self._emit_state()
1184-
1185- # Save our state if the hook completes successfully.
1186- hookenv.atexit(self._save_state)
1187-
1188- # Schedule release of granted locks for the end of the hook.
1189- # This needs to be the last of our atexit callbacks to ensure
1190- # it will be run first when the hook is complete, because there
1191- # is no point mutating our state after it has been saved.
1192- hookenv.atexit(self._release_granted)
1193-
1194- def acquire(self, lock):
1195- '''Acquire the named lock, non-blocking.
1196-
1197- The lock may be granted immediately, or in a future hook.
1198-
1199- Returns True if the lock has been granted. The lock will be
1200- automatically released at the end of the hook in which it is
1201- granted.
1202-
1203- Do not mindlessly call this method, as it triggers a cascade of
1204- hooks. For example, if you call acquire() every time in your
1205- peer relation-changed hook you will end up with an infinite loop
1206- of hooks. It should almost always be guarded by some condition.
1207- '''
1208- unit = hookenv.local_unit()
1209- ts = self.requests[unit].get(lock)
1210- if not ts:
1211- # If there is no outstanding request on the peer relation,
1212- # create one.
1213- self.requests.setdefault(lock, {})
1214- self.requests[unit][lock] = _timestamp()
1215- self.msg('Requested {}'.format(lock))
1216-
1217- # If the leader has granted the lock, yay.
1218- if self.granted(lock):
1219- self.msg('Acquired {}'.format(lock))
1220- return True
1221-
1222- # If the unit making the request also happens to be the
1223- # leader, it must handle the request now. Even though the
1224- # request has been stored on the peer relation, the peer
1225- # relation-changed hook will not be triggered.
1226- if hookenv.is_leader():
1227- return self.grant(lock, unit)
1228-
1229- return False # Can't acquire lock, yet. Maybe next hook.
1230-
1231- def granted(self, lock):
1232- '''Return True if a previously requested lock has been granted'''
1233- unit = hookenv.local_unit()
1234- ts = self.requests[unit].get(lock)
1235- if ts and self.grants.get(unit, {}).get(lock) == ts:
1236- return True
1237- return False
1238-
1239- def requested(self, lock):
1240- '''Return True if we are in the queue for the lock'''
1241- return lock in self.requests[hookenv.local_unit()]
1242-
1243- def request_timestamp(self, lock):
1244- '''Return the timestamp of our outstanding request for lock, or None.
1245-
1246- Returns a datetime.datetime() UTC timestamp, with no tzinfo attribute.
1247- '''
1248- ts = self.requests[hookenv.local_unit()].get(lock, None)
1249- if ts is not None:
1250- return datetime.strptime(ts, _timestamp_format)
1251-
1252- def handle(self):
1253- if not hookenv.is_leader():
1254- return # Only the leader can grant requests.
1255-
1256- self.msg('Leader handling coordinator requests')
1257-
1258- # Clear our grants that have been released.
1259- for unit in self.grants.keys():
1260- for lock, grant_ts in list(self.grants[unit].items()):
1261- req_ts = self.requests.get(unit, {}).get(lock)
1262- if req_ts != grant_ts:
1263- # The request timestamp does not match the granted
1264- # timestamp. Several hooks on 'unit' may have run
1265- # before the leader got a chance to make a decision,
1266- # and 'unit' may have released its lock and attempted
1267- # to reacquire it. This will change the timestamp,
1268- # and we correctly revoke the old grant putting it
1269- # to the end of the queue.
1270- ts = datetime.strptime(self.grants[unit][lock],
1271- _timestamp_format)
1272- del self.grants[unit][lock]
1273- self.released(unit, lock, ts)
1274-
1275- # Grant locks
1276- for unit in self.requests.keys():
1277- for lock in self.requests[unit]:
1278- self.grant(lock, unit)
1279-
1280- def grant(self, lock, unit):
1281- '''Maybe grant the lock to a unit.
1282-
1283- The decision to grant the lock or not is made for $lock
1284- by a corresponding method grant_$lock, which you may define
1285- in a subclass. If no such method is defined, the default_grant
1286- method is used. See Serial.default_grant() for details.
1287- '''
1288- if not hookenv.is_leader():
1289- return False # Not the leader, so we cannot grant.
1290-
1291- # Set of units already granted the lock.
1292- granted = set()
1293- for u in self.grants:
1294- if lock in self.grants[u]:
1295- granted.add(u)
1296- if unit in granted:
1297- return True # Already granted.
1298-
1299- # Ordered list of units waiting for the lock.
1300- reqs = set()
1301- for u in self.requests:
1302- if u in granted:
1303- continue # In the granted set. Not wanted in the req list.
1304- for l, ts in self.requests[u].items():
1305- if l == lock:
1306- reqs.add((ts, u))
1307- queue = [t[1] for t in sorted(reqs)]
1308- if unit not in queue:
1309- return False # Unit has not requested the lock.
1310-
1311- # Locate custom logic, or fallback to the default.
1312- grant_func = getattr(self, 'grant_{}'.format(lock), self.default_grant)
1313-
1314- if grant_func(lock, unit, granted, queue):
1315- # Grant the lock.
1316- self.msg('Leader grants {} to {}'.format(lock, unit))
1317- self.grants.setdefault(unit, {})[lock] = self.requests[unit][lock]
1318- return True
1319-
1320- return False
1321-
1322- def released(self, unit, lock, timestamp):
1323- '''Called on the leader when it has released a lock.
1324-
1325- By default, does nothing but log messages. Override if you
1326- need to perform additional housekeeping when a lock is released,
1327- for example recording timestamps.
1328- '''
1329- interval = _utcnow() - timestamp
1330- self.msg('Leader released {} from {}, held {}'.format(lock, unit,
1331- interval))
1332-
1333- def require(self, lock, guard_func, *guard_args, **guard_kw):
1334- """Decorate a function to be run only when a lock is acquired.
1335-
1336- The lock is requested if the guard function returns True.
1337-
1338- The decorated function is called if the lock has been granted.
1339- """
1340- def decorator(f):
1341- @wraps(f)
1342- def wrapper(*args, **kw):
1343- if self.granted(lock):
1344- self.msg('Granted {}'.format(lock))
1345- return f(*args, **kw)
1346- if guard_func(*guard_args, **guard_kw) and self.acquire(lock):
1347- return f(*args, **kw)
1348- return None
1349- return wrapper
1350- return decorator
1351-
1352- def msg(self, msg):
1353- '''Emit a message. Override to customize log spam.'''
1354- hookenv.log('coordinator.{} {}'.format(self._name(), msg),
1355- level=hookenv.INFO)
1356-
1357- def _name(self):
1358- return self.__class__.__name__
1359-
1360- def _load_state(self):
1361- self.msg('Loading state'.format(self._name()))
1362-
1363- # All responses must be stored in the leadership settings.
1364- # The leader cannot use local state, as a different unit may
1365- # be leader next time. Which is fine, as the leadership
1366- # settings are always available.
1367- self.grants = json.loads(hookenv.leader_get(self.key) or '{}')
1368-
1369- local_unit = hookenv.local_unit()
1370-
1371- # All requests must be stored on the peer relation. This is
1372- # the only channel units have to communicate with the leader.
1373- # Even the leader needs to store its requests here, as a
1374- # different unit may be leader by the time the request can be
1375- # granted.
1376- if self.relid is None:
1377- # The peer relation is not available. Maybe we are early in
1378- # the units's lifecycle. Maybe this unit is standalone.
1379- # Fallback to using local state.
1380- self.msg('No peer relation. Loading local state')
1381- self.requests = {local_unit: self._load_local_state()}
1382- else:
1383- self.requests = self._load_peer_state()
1384- if local_unit not in self.requests:
1385- # The peer relation has just been joined. Update any state
1386- # loaded from our peers with our local state.
1387- self.msg('New peer relation. Merging local state')
1388- self.requests[local_unit] = self._load_local_state()
1389-
1390- def _emit_state(self):
1391- # Emit this units lock status.
1392- for lock in sorted(self.requests[hookenv.local_unit()].keys()):
1393- if self.granted(lock):
1394- self.msg('Granted {}'.format(lock))
1395- else:
1396- self.msg('Waiting on {}'.format(lock))
1397-
1398- def _save_state(self):
1399- self.msg('Publishing state'.format(self._name()))
1400- if hookenv.is_leader():
1401- # sort_keys to ensure stability.
1402- raw = json.dumps(self.grants, sort_keys=True)
1403- hookenv.leader_set({self.key: raw})
1404-
1405- local_unit = hookenv.local_unit()
1406-
1407- if self.relid is None:
1408- # No peer relation yet. Fallback to local state.
1409- self.msg('No peer relation. Saving local state')
1410- self._save_local_state(self.requests[local_unit])
1411- else:
1412- # sort_keys to ensure stability.
1413- raw = json.dumps(self.requests[local_unit], sort_keys=True)
1414- hookenv.relation_set(self.relid, relation_settings={self.key: raw})
1415-
1416- def _load_peer_state(self):
1417- requests = {}
1418- units = set(hookenv.related_units(self.relid))
1419- units.add(hookenv.local_unit())
1420- for unit in units:
1421- raw = hookenv.relation_get(self.key, unit, self.relid)
1422- if raw:
1423- requests[unit] = json.loads(raw)
1424- return requests
1425-
1426- def _local_state_filename(self):
1427- # Include the class name. We allow multiple BaseCoordinator
1428- # subclasses to be instantiated, and they are singletons, so
1429- # this avoids conflicts (unless someone creates and uses two
1430- # BaseCoordinator subclasses with the same class name, so don't
1431- # do that).
1432- return '.charmhelpers.coordinator.{}'.format(self._name())
1433-
1434- def _load_local_state(self):
1435- fn = self._local_state_filename()
1436- if os.path.exists(fn):
1437- with open(fn, 'r') as f:
1438- return json.load(f)
1439- return {}
1440-
1441- def _save_local_state(self, state):
1442- fn = self._local_state_filename()
1443- with open(fn, 'w') as f:
1444- json.dump(state, f)
1445-
1446- def _release_granted(self):
1447- # At the end of every hook, release all locks granted to
1448- # this unit. If a hook neglects to make use of what it
1449- # requested, it will just have to make the request again.
1450- # Implicit release is the only way this will work, as
1451- # if the unit is standalone there may be no future triggers
1452- # called to do a manual release.
1453- unit = hookenv.local_unit()
1454- for lock in list(self.requests[unit].keys()):
1455- if self.granted(lock):
1456- self.msg('Released local {} lock'.format(lock))
1457- del self.requests[unit][lock]
1458-
1459-
1460-class Serial(BaseCoordinator):
1461- def default_grant(self, lock, unit, granted, queue):
1462- '''Default logic to grant a lock to a unit. Unless overridden,
1463- only one unit may hold the lock and it will be granted to the
1464- earliest queued request.
1465-
1466- To define custom logic for $lock, create a subclass and
1467- define a grant_$lock method.
1468-
1469- `unit` is the unit name making the request.
1470-
1471- `granted` is the set of units already granted the lock. It will
1472- never include `unit`. It may be empty.
1473-
1474- `queue` is the list of units waiting for the lock, ordered by time
1475- of request. It will always include `unit`, but `unit` is not
1476- necessarily first.
1477-
1478- Returns True if the lock should be granted to `unit`.
1479- '''
1480- return unit == queue[0] and not granted
1481-
1482-
1483-def _implicit_peer_relation_name():
1484- md = hookenv.metadata()
1485- assert 'peers' in md, 'No peer relations in metadata.yaml'
1486- return sorted(md['peers'].keys())[0]
1487-
1488-
1489-# A human readable, sortable UTC timestamp format.
1490-_timestamp_format = '%Y-%m-%d %H:%M:%S.%fZ'
1491-
1492-
1493-def _utcnow(): # pragma: no cover
1494- # This wrapper exists as mocking datetime methods is problematic.
1495- return datetime.utcnow()
1496-
1497-
1498-def _timestamp():
1499- return _utcnow().strftime(_timestamp_format)
1500diff --git a/hooks/charmhelpers/core/__init__.py b/hooks/charmhelpers/core/__init__.py
1501deleted file mode 100644
1502index d1400a0..0000000
1503--- a/hooks/charmhelpers/core/__init__.py
1504+++ /dev/null
1505@@ -1,15 +0,0 @@
1506-# Copyright 2014-2015 Canonical Limited.
1507-#
1508-# This file is part of charm-helpers.
1509-#
1510-# charm-helpers is free software: you can redistribute it and/or modify
1511-# it under the terms of the GNU Lesser General Public License version 3 as
1512-# published by the Free Software Foundation.
1513-#
1514-# charm-helpers is distributed in the hope that it will be useful,
1515-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1516-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1517-# GNU Lesser General Public License for more details.
1518-#
1519-# You should have received a copy of the GNU Lesser General Public License
1520-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1521diff --git a/hooks/charmhelpers/core/decorators.py b/hooks/charmhelpers/core/decorators.py
1522deleted file mode 100644
1523index bb05620..0000000
1524--- a/hooks/charmhelpers/core/decorators.py
1525+++ /dev/null
1526@@ -1,57 +0,0 @@
1527-# Copyright 2014-2015 Canonical Limited.
1528-#
1529-# This file is part of charm-helpers.
1530-#
1531-# charm-helpers is free software: you can redistribute it and/or modify
1532-# it under the terms of the GNU Lesser General Public License version 3 as
1533-# published by the Free Software Foundation.
1534-#
1535-# charm-helpers is distributed in the hope that it will be useful,
1536-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1537-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1538-# GNU Lesser General Public License for more details.
1539-#
1540-# You should have received a copy of the GNU Lesser General Public License
1541-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1542-
1543-#
1544-# Copyright 2014 Canonical Ltd.
1545-#
1546-# Authors:
1547-# Edward Hope-Morley <opentastic@gmail.com>
1548-#
1549-
1550-import time
1551-
1552-from charmhelpers.core.hookenv import (
1553- log,
1554- INFO,
1555-)
1556-
1557-
1558-def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
1559- """If the decorated function raises exception exc_type, allow num_retries
1560- retry attempts before raise the exception.
1561- """
1562- def _retry_on_exception_inner_1(f):
1563- def _retry_on_exception_inner_2(*args, **kwargs):
1564- retries = num_retries
1565- multiplier = 1
1566- while True:
1567- try:
1568- return f(*args, **kwargs)
1569- except exc_type:
1570- if not retries:
1571- raise
1572-
1573- delay = base_delay * multiplier
1574- multiplier += 1
1575- log("Retrying '%s' %d more times (delay=%s)" %
1576- (f.__name__, retries, delay), level=INFO)
1577- retries -= 1
1578- if delay:
1579- time.sleep(delay)
1580-
1581- return _retry_on_exception_inner_2
1582-
1583- return _retry_on_exception_inner_1
1584diff --git a/hooks/charmhelpers/core/files.py b/hooks/charmhelpers/core/files.py
1585deleted file mode 100644
1586index 0f12d32..0000000
1587--- a/hooks/charmhelpers/core/files.py
1588+++ /dev/null
1589@@ -1,45 +0,0 @@
1590-#!/usr/bin/env python
1591-# -*- coding: utf-8 -*-
1592-
1593-# Copyright 2014-2015 Canonical Limited.
1594-#
1595-# This file is part of charm-helpers.
1596-#
1597-# charm-helpers is free software: you can redistribute it and/or modify
1598-# it under the terms of the GNU Lesser General Public License version 3 as
1599-# published by the Free Software Foundation.
1600-#
1601-# charm-helpers is distributed in the hope that it will be useful,
1602-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1603-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1604-# GNU Lesser General Public License for more details.
1605-#
1606-# You should have received a copy of the GNU Lesser General Public License
1607-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1608-
1609-__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'
1610-
1611-import os
1612-import subprocess
1613-
1614-
1615-def sed(filename, before, after, flags='g'):
1616- """
1617- Search and replaces the given pattern on filename.
1618-
1619- :param filename: relative or absolute file path.
1620- :param before: expression to be replaced (see 'man sed')
1621- :param after: expression to replace with (see 'man sed')
1622- :param flags: sed-compatible regex flags in example, to make
1623- the search and replace case insensitive, specify ``flags="i"``.
1624- The ``g`` flag is always specified regardless, so you do not
1625- need to remember to include it when overriding this parameter.
1626- :returns: If the sed command exit code was zero then return,
1627- otherwise raise CalledProcessError.
1628- """
1629- expression = r's/{0}/{1}/{2}'.format(before,
1630- after, flags)
1631-
1632- return subprocess.check_call(["sed", "-i", "-r", "-e",
1633- expression,
1634- os.path.expanduser(filename)])
1635diff --git a/hooks/charmhelpers/core/fstab.py b/hooks/charmhelpers/core/fstab.py
1636deleted file mode 100644
1637index 3056fba..0000000
1638--- a/hooks/charmhelpers/core/fstab.py
1639+++ /dev/null
1640@@ -1,134 +0,0 @@
1641-#!/usr/bin/env python
1642-# -*- coding: utf-8 -*-
1643-
1644-# Copyright 2014-2015 Canonical Limited.
1645-#
1646-# This file is part of charm-helpers.
1647-#
1648-# charm-helpers is free software: you can redistribute it and/or modify
1649-# it under the terms of the GNU Lesser General Public License version 3 as
1650-# published by the Free Software Foundation.
1651-#
1652-# charm-helpers is distributed in the hope that it will be useful,
1653-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1654-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1655-# GNU Lesser General Public License for more details.
1656-#
1657-# You should have received a copy of the GNU Lesser General Public License
1658-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1659-
1660-import io
1661-import os
1662-
1663-__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1664-
1665-
1666-class Fstab(io.FileIO):
1667- """This class extends file in order to implement a file reader/writer
1668- for file `/etc/fstab`
1669- """
1670-
1671- class Entry(object):
1672- """Entry class represents a non-comment line on the `/etc/fstab` file
1673- """
1674- def __init__(self, device, mountpoint, filesystem,
1675- options, d=0, p=0):
1676- self.device = device
1677- self.mountpoint = mountpoint
1678- self.filesystem = filesystem
1679-
1680- if not options:
1681- options = "defaults"
1682-
1683- self.options = options
1684- self.d = int(d)
1685- self.p = int(p)
1686-
1687- def __eq__(self, o):
1688- return str(self) == str(o)
1689-
1690- def __str__(self):
1691- return "{} {} {} {} {} {}".format(self.device,
1692- self.mountpoint,
1693- self.filesystem,
1694- self.options,
1695- self.d,
1696- self.p)
1697-
1698- DEFAULT_PATH = os.path.join(os.path.sep, 'etc', 'fstab')
1699-
1700- def __init__(self, path=None):
1701- if path:
1702- self._path = path
1703- else:
1704- self._path = self.DEFAULT_PATH
1705- super(Fstab, self).__init__(self._path, 'rb+')
1706-
1707- def _hydrate_entry(self, line):
1708- # NOTE: use split with no arguments to split on any
1709- # whitespace including tabs
1710- return Fstab.Entry(*filter(
1711- lambda x: x not in ('', None),
1712- line.strip("\n").split()))
1713-
1714- @property
1715- def entries(self):
1716- self.seek(0)
1717- for line in self.readlines():
1718- line = line.decode('us-ascii')
1719- try:
1720- if line.strip() and not line.strip().startswith("#"):
1721- yield self._hydrate_entry(line)
1722- except ValueError:
1723- pass
1724-
1725- def get_entry_by_attr(self, attr, value):
1726- for entry in self.entries:
1727- e_attr = getattr(entry, attr)
1728- if e_attr == value:
1729- return entry
1730- return None
1731-
1732- def add_entry(self, entry):
1733- if self.get_entry_by_attr('device', entry.device):
1734- return False
1735-
1736- self.write((str(entry) + '\n').encode('us-ascii'))
1737- self.truncate()
1738- return entry
1739-
1740- def remove_entry(self, entry):
1741- self.seek(0)
1742-
1743- lines = [l.decode('us-ascii') for l in self.readlines()]
1744-
1745- found = False
1746- for index, line in enumerate(lines):
1747- if line.strip() and not line.strip().startswith("#"):
1748- if self._hydrate_entry(line) == entry:
1749- found = True
1750- break
1751-
1752- if not found:
1753- return False
1754-
1755- lines.remove(line)
1756-
1757- self.seek(0)
1758- self.write(''.join(lines).encode('us-ascii'))
1759- self.truncate()
1760- return True
1761-
1762- @classmethod
1763- def remove_by_mountpoint(cls, mountpoint, path=None):
1764- fstab = cls(path=path)
1765- entry = fstab.get_entry_by_attr('mountpoint', mountpoint)
1766- if entry:
1767- return fstab.remove_entry(entry)
1768- return False
1769-
1770- @classmethod
1771- def add(cls, device, mountpoint, filesystem, options=None, path=None):
1772- return cls(path=path).add_entry(Fstab.Entry(device,
1773- mountpoint, filesystem,
1774- options=options))
1775diff --git a/hooks/charmhelpers/core/hookenv.py b/hooks/charmhelpers/core/hookenv.py
1776deleted file mode 100644
1777index 84a640d..0000000
1778--- a/hooks/charmhelpers/core/hookenv.py
1779+++ /dev/null
1780@@ -1,816 +0,0 @@
1781-# Copyright 2014-2015 Canonical Limited.
1782-#
1783-# This file is part of charm-helpers.
1784-#
1785-# charm-helpers is free software: you can redistribute it and/or modify
1786-# it under the terms of the GNU Lesser General Public License version 3 as
1787-# published by the Free Software Foundation.
1788-#
1789-# charm-helpers is distributed in the hope that it will be useful,
1790-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1791-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1792-# GNU Lesser General Public License for more details.
1793-#
1794-# You should have received a copy of the GNU Lesser General Public License
1795-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1796-
1797-"Interactions with the Juju environment"
1798-# Copyright 2013 Canonical Ltd.
1799-#
1800-# Authors:
1801-# Charm Helpers Developers <juju@lists.ubuntu.com>
1802-
1803-from __future__ import print_function
1804-import copy
1805-from distutils.version import LooseVersion
1806-from functools import wraps
1807-import glob
1808-import os
1809-import json
1810-import yaml
1811-import subprocess
1812-import sys
1813-import errno
1814-import tempfile
1815-from subprocess import CalledProcessError
1816-
1817-import six
1818-if not six.PY3:
1819- from UserDict import UserDict
1820-else:
1821- from collections import UserDict
1822-
1823-CRITICAL = "CRITICAL"
1824-ERROR = "ERROR"
1825-WARNING = "WARNING"
1826-INFO = "INFO"
1827-DEBUG = "DEBUG"
1828-MARKER = object()
1829-
1830-cache = {}
1831-
1832-
1833-def cached(func):
1834- """Cache return values for multiple executions of func + args
1835-
1836- For example::
1837-
1838- @cached
1839- def unit_get(attribute):
1840- pass
1841-
1842- unit_get('test')
1843-
1844- will cache the result of unit_get + 'test' for future calls.
1845- """
1846- @wraps(func)
1847- def wrapper(*args, **kwargs):
1848- global cache
1849- key = str((func, args, kwargs))
1850- try:
1851- return cache[key]
1852- except KeyError:
1853- pass # Drop out of the exception handler scope.
1854- res = func(*args, **kwargs)
1855- cache[key] = res
1856- return res
1857- return wrapper
1858-
1859-
1860-def flush(key):
1861- """Flushes any entries from function cache where the
1862- key is found in the function+args """
1863- flush_list = []
1864- for item in cache:
1865- if key in item:
1866- flush_list.append(item)
1867- for item in flush_list:
1868- del cache[item]
1869-
1870-
1871-def log(message, level=None):
1872- """Write a message to the juju log"""
1873- command = ['juju-log']
1874- if level:
1875- command += ['-l', level]
1876- if not isinstance(message, six.string_types):
1877- message = repr(message)
1878- command += [message]
1879- # Missing juju-log should not cause failures in unit tests
1880- # Send log output to stderr
1881- try:
1882- subprocess.call(command)
1883- except OSError as e:
1884- if e.errno == errno.ENOENT:
1885- if level:
1886- message = "{}: {}".format(level, message)
1887- message = "juju-log: {}".format(message)
1888- print(message, file=sys.stderr)
1889- else:
1890- raise
1891-
1892-
1893-class Serializable(UserDict):
1894- """Wrapper, an object that can be serialized to yaml or json"""
1895-
1896- def __init__(self, obj):
1897- # wrap the object
1898- UserDict.__init__(self)
1899- self.data = obj
1900-
1901- def __getattr__(self, attr):
1902- # See if this object has attribute.
1903- if attr in ("json", "yaml", "data"):
1904- return self.__dict__[attr]
1905- # Check for attribute in wrapped object.
1906- got = getattr(self.data, attr, MARKER)
1907- if got is not MARKER:
1908- return got
1909- # Proxy to the wrapped object via dict interface.
1910- try:
1911- return self.data[attr]
1912- except KeyError:
1913- raise AttributeError(attr)
1914-
1915- def __getstate__(self):
1916- # Pickle as a standard dictionary.
1917- return self.data
1918-
1919- def __setstate__(self, state):
1920- # Unpickle into our wrapper.
1921- self.data = state
1922-
1923- def json(self):
1924- """Serialize the object to json"""
1925- return json.dumps(self.data)
1926-
1927- def yaml(self):
1928- """Serialize the object to yaml"""
1929- return yaml.dump(self.data)
1930-
1931-
1932-def execution_environment():
1933- """A convenient bundling of the current execution context"""
1934- context = {}
1935- context['conf'] = config()
1936- if relation_id():
1937- context['reltype'] = relation_type()
1938- context['relid'] = relation_id()
1939- context['rel'] = relation_get()
1940- context['unit'] = local_unit()
1941- context['rels'] = relations()
1942- context['env'] = os.environ
1943- return context
1944-
1945-
1946-def in_relation_hook():
1947- """Determine whether we're running in a relation hook"""
1948- return 'JUJU_RELATION' in os.environ
1949-
1950-
1951-def relation_type():
1952- """The scope for the current relation hook"""
1953- return os.environ.get('JUJU_RELATION', None)
1954-
1955-
1956-def relation_id():
1957- """The relation ID for the current relation hook"""
1958- return os.environ.get('JUJU_RELATION_ID', None)
1959-
1960-
1961-def local_unit():
1962- """Local unit ID"""
1963- return os.environ['JUJU_UNIT_NAME']
1964-
1965-
1966-def remote_unit():
1967- """The remote unit for the current relation hook"""
1968- return os.environ.get('JUJU_REMOTE_UNIT', None)
1969-
1970-
1971-def service_name():
1972- """The name service group this unit belongs to"""
1973- return local_unit().split('/')[0]
1974-
1975-
1976-def hook_name():
1977- """The name of the currently executing hook"""
1978- return os.path.basename(sys.argv[0])
1979-
1980-
1981-class Config(dict):
1982- """A dictionary representation of the charm's config.yaml, with some
1983- extra features:
1984-
1985- - See which values in the dictionary have changed since the previous hook.
1986- - For values that have changed, see what the previous value was.
1987- - Store arbitrary data for use in a later hook.
1988-
1989- NOTE: Do not instantiate this object directly - instead call
1990- ``hookenv.config()``, which will return an instance of :class:`Config`.
1991-
1992- Example usage::
1993-
1994- >>> # inside a hook
1995- >>> from charmhelpers.core import hookenv
1996- >>> config = hookenv.config()
1997- >>> config['foo']
1998- 'bar'
1999- >>> # store a new key/value for later use
2000- >>> config['mykey'] = 'myval'
2001-
2002-
2003- >>> # user runs `juju set mycharm foo=baz`
2004- >>> # now we're inside subsequent config-changed hook
2005- >>> config = hookenv.config()
2006- >>> config['foo']
2007- 'baz'
2008- >>> # test to see if this val has changed since last hook
2009- >>> config.changed('foo')
2010- True
2011- >>> # what was the previous value?
2012- >>> config.previous('foo')
2013- 'bar'
2014- >>> # keys/values that we add are preserved across hooks
2015- >>> config['mykey']
2016- 'myval'
2017-
2018- """
2019- CONFIG_FILE_NAME = '.juju-persistent-config'
2020-
2021- def __init__(self, *args, **kw):
2022- super(Config, self).__init__(*args, **kw)
2023- self.implicit_save = True
2024- self._prev_dict = None
2025- self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
2026- if os.path.exists(self.path):
2027- self.load_previous()
2028- atexit(self._implicit_save)
2029-
2030- def load_previous(self, path=None):
2031- """Load previous copy of config from disk.
2032-
2033- In normal usage you don't need to call this method directly - it
2034- is called automatically at object initialization.
2035-
2036- :param path:
2037-
2038- File path from which to load the previous config. If `None`,
2039- config is loaded from the default location. If `path` is
2040- specified, subsequent `save()` calls will write to the same
2041- path.
2042-
2043- """
2044- self.path = path or self.path
2045- with open(self.path) as f:
2046- self._prev_dict = json.load(f)
2047- for k, v in copy.deepcopy(self._prev_dict).items():
2048- if k not in self:
2049- self[k] = v
2050-
2051- def changed(self, key):
2052- """Return True if the current value for this key is different from
2053- the previous value.
2054-
2055- """
2056- if self._prev_dict is None:
2057- return True
2058- return self.previous(key) != self.get(key)
2059-
2060- def previous(self, key):
2061- """Return previous value for this key, or None if there
2062- is no previous value.
2063-
2064- """
2065- if self._prev_dict:
2066- return self._prev_dict.get(key)
2067- return None
2068-
2069- def save(self):
2070- """Save this config to disk.
2071-
2072- If the charm is using the :mod:`Services Framework <services.base>`
2073- or :meth:'@hook <Hooks.hook>' decorator, this
2074- is called automatically at the end of successful hook execution.
2075- Otherwise, it should be called directly by user code.
2076-
2077- To disable automatic saves, set ``implicit_save=False`` on this
2078- instance.
2079-
2080- """
2081- with open(self.path, 'w') as f:
2082- json.dump(self, f)
2083-
2084- def _implicit_save(self):
2085- if self.implicit_save:
2086- self.save()
2087-
2088-
2089-@cached
2090-def config(scope=None):
2091- """Juju charm configuration"""
2092- config_cmd_line = ['config-get']
2093- if scope is not None:
2094- config_cmd_line.append(scope)
2095- config_cmd_line.append('--format=json')
2096- try:
2097- config_data = json.loads(
2098- subprocess.check_output(config_cmd_line).decode('UTF-8'))
2099- if scope is not None:
2100- return config_data
2101- return Config(config_data)
2102- except ValueError:
2103- return None
2104-
2105-
2106-@cached
2107-def relation_get(attribute=None, unit=None, rid=None):
2108- """Get relation information"""
2109- _args = ['relation-get', '--format=json']
2110- if rid:
2111- _args.append('-r')
2112- _args.append(rid)
2113- _args.append(attribute or '-')
2114- if unit:
2115- _args.append(unit)
2116- try:
2117- return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2118- except ValueError:
2119- return None
2120- except CalledProcessError as e:
2121- if e.returncode == 2:
2122- return None
2123- raise
2124-
2125-
2126-def relation_set(relation_id=None, relation_settings=None, **kwargs):
2127- """Set relation information for the current unit"""
2128- relation_settings = relation_settings if relation_settings else {}
2129- relation_cmd_line = ['relation-set']
2130- accepts_file = "--file" in subprocess.check_output(
2131- relation_cmd_line + ["--help"], universal_newlines=True)
2132- if relation_id is not None:
2133- relation_cmd_line.extend(('-r', relation_id))
2134- settings = relation_settings.copy()
2135- settings.update(kwargs)
2136- for key, value in settings.items():
2137- # Force value to be a string: it always should, but some call
2138- # sites pass in things like dicts or numbers.
2139- if value is not None:
2140- settings[key] = "{}".format(value)
2141- if accepts_file:
2142- # --file was introduced in Juju 1.23.2. Use it by default if
2143- # available, since otherwise we'll break if the relation data is
2144- # too big. Ideally we should tell relation-set to read the data from
2145- # stdin, but that feature is broken in 1.23.2: Bug #1454678.
2146- with tempfile.NamedTemporaryFile(delete=False) as settings_file:
2147- settings_file.write(yaml.safe_dump(settings).encode("utf-8"))
2148- subprocess.check_call(
2149- relation_cmd_line + ["--file", settings_file.name])
2150- os.remove(settings_file.name)
2151- else:
2152- for key, value in settings.items():
2153- if value is None:
2154- relation_cmd_line.append('{}='.format(key))
2155- else:
2156- relation_cmd_line.append('{}={}'.format(key, value))
2157- subprocess.check_call(relation_cmd_line)
2158- # Flush cache of any relation-gets for local unit
2159- flush(local_unit())
2160-
2161-
2162-def relation_clear(r_id=None):
2163- ''' Clears any relation data already set on relation r_id '''
2164- settings = relation_get(rid=r_id,
2165- unit=local_unit())
2166- for setting in settings:
2167- if setting not in ['public-address', 'private-address']:
2168- settings[setting] = None
2169- relation_set(relation_id=r_id,
2170- **settings)
2171-
2172-
2173-@cached
2174-def relation_ids(reltype=None):
2175- """A list of relation_ids"""
2176- reltype = reltype or relation_type()
2177- relid_cmd_line = ['relation-ids', '--format=json']
2178- if reltype is not None:
2179- relid_cmd_line.append(reltype)
2180- return json.loads(
2181- subprocess.check_output(relid_cmd_line).decode('UTF-8')) or []
2182- return []
2183-
2184-
2185-@cached
2186-def related_units(relid=None):
2187- """A list of related units"""
2188- relid = relid or relation_id()
2189- units_cmd_line = ['relation-list', '--format=json']
2190- if relid is not None:
2191- units_cmd_line.extend(('-r', relid))
2192- return json.loads(
2193- subprocess.check_output(units_cmd_line).decode('UTF-8')) or []
2194-
2195-
2196-@cached
2197-def relation_for_unit(unit=None, rid=None):
2198- """Get the json represenation of a unit's relation"""
2199- unit = unit or remote_unit()
2200- relation = relation_get(unit=unit, rid=rid)
2201- for key in relation:
2202- if key.endswith('-list'):
2203- relation[key] = relation[key].split()
2204- relation['__unit__'] = unit
2205- return relation
2206-
2207-
2208-@cached
2209-def relations_for_id(relid=None):
2210- """Get relations of a specific relation ID"""
2211- relation_data = []
2212- relid = relid or relation_ids()
2213- for unit in related_units(relid):
2214- unit_data = relation_for_unit(unit, relid)
2215- unit_data['__relid__'] = relid
2216- relation_data.append(unit_data)
2217- return relation_data
2218-
2219-
2220-@cached
2221-def relations_of_type(reltype=None):
2222- """Get relations of a specific type"""
2223- relation_data = []
2224- reltype = reltype or relation_type()
2225- for relid in relation_ids(reltype):
2226- for relation in relations_for_id(relid):
2227- relation['__relid__'] = relid
2228- relation_data.append(relation)
2229- return relation_data
2230-
2231-
2232-@cached
2233-def metadata():
2234- """Get the current charm metadata.yaml contents as a python object"""
2235- with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
2236- return yaml.safe_load(md)
2237-
2238-
2239-@cached
2240-def relation_types():
2241- """Get a list of relation types supported by this charm"""
2242- rel_types = []
2243- md = metadata()
2244- for key in ('provides', 'requires', 'peers'):
2245- section = md.get(key)
2246- if section:
2247- rel_types.extend(section.keys())
2248- return rel_types
2249-
2250-
2251-@cached
2252-def peer_relation_id():
2253- '''Get a peer relation id if a peer relation has been joined, else None.'''
2254- md = metadata()
2255- section = md.get('peers')
2256- if section:
2257- for key in section:
2258- relids = relation_ids(key)
2259- if relids:
2260- return relids[0]
2261- return None
2262-
2263-
2264-@cached
2265-def charm_name():
2266- """Get the name of the current charm as is specified on metadata.yaml"""
2267- return metadata().get('name')
2268-
2269-
2270-@cached
2271-def relations():
2272- """Get a nested dictionary of relation data for all related units"""
2273- rels = {}
2274- for reltype in relation_types():
2275- relids = {}
2276- for relid in relation_ids(reltype):
2277- units = {local_unit(): relation_get(unit=local_unit(), rid=relid)}
2278- for unit in related_units(relid):
2279- reldata = relation_get(unit=unit, rid=relid)
2280- units[unit] = reldata
2281- relids[relid] = units
2282- rels[reltype] = relids
2283- return rels
2284-
2285-
2286-@cached
2287-def is_relation_made(relation, keys='private-address'):
2288- '''
2289- Determine whether a relation is established by checking for
2290- presence of key(s). If a list of keys is provided, they
2291- must all be present for the relation to be identified as made
2292- '''
2293- if isinstance(keys, str):
2294- keys = [keys]
2295- for r_id in relation_ids(relation):
2296- for unit in related_units(r_id):
2297- context = {}
2298- for k in keys:
2299- context[k] = relation_get(k, rid=r_id,
2300- unit=unit)
2301- if None not in context.values():
2302- return True
2303- return False
2304-
2305-
2306-def open_port(port, protocol="TCP"):
2307- """Open a service network port"""
2308- _args = ['open-port']
2309- _args.append('{}/{}'.format(port, protocol))
2310- subprocess.check_call(_args)
2311-
2312-
2313-def close_port(port, protocol="TCP"):
2314- """Close a service network port"""
2315- _args = ['close-port']
2316- _args.append('{}/{}'.format(port, protocol))
2317- subprocess.check_call(_args)
2318-
2319-
2320-@cached
2321-def unit_get(attribute):
2322- """Get the unit ID for the remote unit"""
2323- _args = ['unit-get', '--format=json', attribute]
2324- try:
2325- return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2326- except ValueError:
2327- return None
2328-
2329-
2330-def unit_public_ip():
2331- """Get this unit's public IP address"""
2332- return unit_get('public-address')
2333-
2334-
2335-def unit_private_ip():
2336- """Get this unit's private IP address"""
2337- return unit_get('private-address')
2338-
2339-
2340-class UnregisteredHookError(Exception):
2341- """Raised when an undefined hook is called"""
2342- pass
2343-
2344-
2345-class Hooks(object):
2346- """A convenient handler for hook functions.
2347-
2348- Example::
2349-
2350- hooks = Hooks()
2351-
2352- # register a hook, taking its name from the function name
2353- @hooks.hook()
2354- def install():
2355- pass # your code here
2356-
2357- # register a hook, providing a custom hook name
2358- @hooks.hook("config-changed")
2359- def config_changed():
2360- pass # your code here
2361-
2362- if __name__ == "__main__":
2363- # execute a hook based on the name the program is called by
2364- hooks.execute(sys.argv)
2365- """
2366-
2367- def __init__(self, config_save=None):
2368- super(Hooks, self).__init__()
2369- self._hooks = {}
2370-
2371- # For unknown reasons, we allow the Hooks constructor to override
2372- # config().implicit_save.
2373- if config_save is not None:
2374- config().implicit_save = config_save
2375-
2376- def register(self, name, function):
2377- """Register a hook"""
2378- self._hooks[name] = function
2379-
2380- def execute(self, args):
2381- """Execute a registered hook based on args[0]"""
2382- _run_atstart()
2383- hook_name = os.path.basename(args[0])
2384- if hook_name in self._hooks:
2385- try:
2386- self._hooks[hook_name]()
2387- except SystemExit as x:
2388- if x.code is None or x.code == 0:
2389- _run_atexit()
2390- raise
2391- _run_atexit()
2392- else:
2393- raise UnregisteredHookError(hook_name)
2394-
2395- def hook(self, *hook_names):
2396- """Decorator, registering them as hooks"""
2397- def wrapper(decorated):
2398- for hook_name in hook_names:
2399- self.register(hook_name, decorated)
2400- else:
2401- self.register(decorated.__name__, decorated)
2402- if '_' in decorated.__name__:
2403- self.register(
2404- decorated.__name__.replace('_', '-'), decorated)
2405- return decorated
2406- return wrapper
2407-
2408-
2409-def charm_dir():
2410- """Return the root directory of the current charm"""
2411- return os.environ.get('CHARM_DIR')
2412-
2413-
2414-@cached
2415-def action_get(key=None):
2416- """Gets the value of an action parameter, or all key/value param pairs"""
2417- cmd = ['action-get']
2418- if key is not None:
2419- cmd.append(key)
2420- cmd.append('--format=json')
2421- action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
2422- return action_data
2423-
2424-
2425-def action_set(values):
2426- """Sets the values to be returned after the action finishes"""
2427- cmd = ['action-set']
2428- for k, v in list(values.items()):
2429- cmd.append('{}={}'.format(k, v))
2430- subprocess.check_call(cmd)
2431-
2432-
2433-def action_fail(message):
2434- """Sets the action status to failed and sets the error message.
2435-
2436- The results set by action_set are preserved."""
2437- subprocess.check_call(['action-fail', message])
2438-
2439-
2440-def status_set(workload_state, message):
2441- """Set the workload state with a message
2442-
2443- Use status-set to set the workload state with a message which is visible
2444- to the user via juju status. If the status-set command is not found then
2445- assume this is juju < 1.23 and juju-log the message unstead.
2446-
2447- workload_state -- valid juju workload state.
2448- message -- status update message
2449- """
2450- valid_states = ['maintenance', 'blocked', 'waiting', 'active']
2451- if workload_state not in valid_states:
2452- raise ValueError(
2453- '{!r} is not a valid workload state'.format(workload_state)
2454- )
2455- cmd = ['status-set', workload_state, message]
2456- try:
2457- ret = subprocess.call(cmd)
2458- if ret == 0:
2459- return
2460- except OSError as e:
2461- if e.errno != errno.ENOENT:
2462- raise
2463- log_message = 'status-set failed: {} {}'.format(workload_state,
2464- message)
2465- log(log_message, level='INFO')
2466-
2467-
2468-def status_get():
2469- """Retrieve the previously set juju workload state
2470-
2471- If the status-set command is not found then assume this is juju < 1.23 and
2472- return 'unknown'
2473- """
2474- cmd = ['status-get']
2475- try:
2476- raw_status = subprocess.check_output(cmd, universal_newlines=True)
2477- status = raw_status.rstrip()
2478- return status
2479- except OSError as e:
2480- if e.errno == errno.ENOENT:
2481- return 'unknown'
2482- else:
2483- raise
2484-
2485-
2486-def translate_exc(from_exc, to_exc):
2487- def inner_translate_exc1(f):
2488- @wraps(f)
2489- def inner_translate_exc2(*args, **kwargs):
2490- try:
2491- return f(*args, **kwargs)
2492- except from_exc:
2493- raise to_exc
2494-
2495- return inner_translate_exc2
2496-
2497- return inner_translate_exc1
2498-
2499-
2500-@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
2501-def is_leader():
2502- """Does the current unit hold the juju leadership
2503-
2504- Uses juju to determine whether the current unit is the leader of its peers
2505- """
2506- cmd = ['is-leader', '--format=json']
2507- return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
2508-
2509-
2510-@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
2511-def leader_get(attribute=None):
2512- """Juju leader get value(s)"""
2513- cmd = ['leader-get', '--format=json'] + [attribute or '-']
2514- return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
2515-
2516-
2517-@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
2518-def leader_set(settings=None, **kwargs):
2519- """Juju leader set value(s)"""
2520- # Don't log secrets.
2521- # log("Juju leader-set '%s'" % (settings), level=DEBUG)
2522- cmd = ['leader-set']
2523- settings = settings or {}
2524- settings.update(kwargs)
2525- for k, v in settings.items():
2526- if v is None:
2527- cmd.append('{}='.format(k))
2528- else:
2529- cmd.append('{}={}'.format(k, v))
2530- subprocess.check_call(cmd)
2531-
2532-
2533-@cached
2534-def juju_version():
2535- """Full version string (eg. '1.23.3.1-trusty-amd64')"""
2536- # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
2537- jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
2538- return subprocess.check_output([jujud, 'version'],
2539- universal_newlines=True).strip()
2540-
2541-
2542-@cached
2543-def has_juju_version(minimum_version):
2544- """Return True if the Juju version is at least the provided version"""
2545- return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
2546-
2547-
2548-_atexit = []
2549-_atstart = []
2550-
2551-
2552-def atstart(callback, *args, **kwargs):
2553- '''Schedule a callback to run before the main hook.
2554-
2555- Callbacks are run in the order they were added.
2556-
2557- This is useful for modules and classes to perform initialization
2558- and inject behavior. In particular:
2559-
2560- - Run common code before all of your hooks, such as logging
2561- the hook name or interesting relation data.
2562- - Defer object or module initialization that requires a hook
2563- context until we know there actually is a hook context,
2564- making testing easier.
2565- - Rather than requiring charm authors to include boilerplate to
2566- invoke your helper's behavior, have it run automatically if
2567- your object is instantiated or module imported.
2568-
2569- This is not at all useful after your hook framework as been launched.
2570- '''
2571- global _atstart
2572- _atstart.append((callback, args, kwargs))
2573-
2574-
2575-def atexit(callback, *args, **kwargs):
2576- '''Schedule a callback to run on successful hook completion.
2577-
2578- Callbacks are run in the reverse order that they were added.'''
2579- _atexit.append((callback, args, kwargs))
2580-
2581-
2582-def _run_atstart():
2583- '''Hook frameworks must invoke this before running the main hook body.'''
2584- global _atstart
2585- for callback, args, kwargs in _atstart:
2586- callback(*args, **kwargs)
2587- del _atstart[:]
2588-
2589-
2590-def _run_atexit():
2591- '''Hook frameworks must invoke this after the main hook body has
2592- successfully completed. Do not invoke it if the hook fails.'''
2593- global _atexit
2594- for callback, args, kwargs in reversed(_atexit):
2595- callback(*args, **kwargs)
2596- del _atexit[:]
2597diff --git a/hooks/charmhelpers/core/host.py b/hooks/charmhelpers/core/host.py
2598deleted file mode 100644
2599index 9e6b674..0000000
2600--- a/hooks/charmhelpers/core/host.py
2601+++ /dev/null
2602@@ -1,510 +0,0 @@
2603-# Copyright 2014-2015 Canonical Limited.
2604-#
2605-# This file is part of charm-helpers.
2606-#
2607-# charm-helpers is free software: you can redistribute it and/or modify
2608-# it under the terms of the GNU Lesser General Public License version 3 as
2609-# published by the Free Software Foundation.
2610-#
2611-# charm-helpers is distributed in the hope that it will be useful,
2612-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2613-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2614-# GNU Lesser General Public License for more details.
2615-#
2616-# You should have received a copy of the GNU Lesser General Public License
2617-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2618-
2619-"""Tools for working with the host system"""
2620-# Copyright 2012 Canonical Ltd.
2621-#
2622-# Authors:
2623-# Nick Moffitt <nick.moffitt@canonical.com>
2624-# Matthew Wedgwood <matthew.wedgwood@canonical.com>
2625-
2626-import os
2627-import re
2628-import pwd
2629-import glob
2630-import grp
2631-import random
2632-import string
2633-import subprocess
2634-import hashlib
2635-from contextlib import contextmanager
2636-from collections import OrderedDict
2637-
2638-import six
2639-
2640-from .hookenv import log
2641-from .fstab import Fstab
2642-
2643-
2644-def service_start(service_name):
2645- """Start a system service"""
2646- return service('start', service_name)
2647-
2648-
2649-def service_stop(service_name):
2650- """Stop a system service"""
2651- return service('stop', service_name)
2652-
2653-
2654-def service_restart(service_name):
2655- """Restart a system service"""
2656- return service('restart', service_name)
2657-
2658-
2659-def service_reload(service_name, restart_on_failure=False):
2660- """Reload a system service, optionally falling back to restart if
2661- reload fails"""
2662- service_result = service('reload', service_name)
2663- if not service_result and restart_on_failure:
2664- service_result = service('restart', service_name)
2665- return service_result
2666-
2667-
2668-def service_pause(service_name, init_dir=None):
2669- """Pause a system service.
2670-
2671- Stop it, and prevent it from starting again at boot."""
2672- if init_dir is None:
2673- init_dir = "/etc/init"
2674- stopped = service_stop(service_name)
2675- # XXX: Support systemd too
2676- override_path = os.path.join(
2677- init_dir, '{}.conf.override'.format(service_name))
2678- with open(override_path, 'w') as fh:
2679- fh.write("manual\n")
2680- return stopped
2681-
2682-
2683-def service_resume(service_name, init_dir=None):
2684- """Resume a system service.
2685-
2686- Reenable starting again at boot. Start the service"""
2687- # XXX: Support systemd too
2688- if init_dir is None:
2689- init_dir = "/etc/init"
2690- override_path = os.path.join(
2691- init_dir, '{}.conf.override'.format(service_name))
2692- if os.path.exists(override_path):
2693- os.unlink(override_path)
2694- started = service_start(service_name)
2695- return started
2696-
2697-
2698-def service(action, service_name):
2699- """Control a system service"""
2700- cmd = ['service', service_name, action]
2701- return subprocess.call(cmd) == 0
2702-
2703-
2704-def service_running(service):
2705- """Determine whether a system service is running"""
2706- try:
2707- output = subprocess.check_output(
2708- ['service', service, 'status'],
2709- stderr=subprocess.STDOUT).decode('UTF-8')
2710- except subprocess.CalledProcessError:
2711- return False
2712- else:
2713- if ("start/running" in output or "is running" in output):
2714- return True
2715- else:
2716- return False
2717-
2718-
2719-def service_available(service_name):
2720- """Determine whether a system service is available"""
2721- try:
2722- subprocess.check_output(
2723- ['service', service_name, 'status'],
2724- stderr=subprocess.STDOUT).decode('UTF-8')
2725- except subprocess.CalledProcessError as e:
2726- return b'unrecognized service' not in e.output
2727- else:
2728- return True
2729-
2730-
2731-def adduser(username, password=None, shell='/bin/bash', system_user=False):
2732- """Add a user to the system"""
2733- try:
2734- user_info = pwd.getpwnam(username)
2735- log('user {0} already exists!'.format(username))
2736- except KeyError:
2737- log('creating user {0}'.format(username))
2738- cmd = ['useradd']
2739- if system_user or password is None:
2740- cmd.append('--system')
2741- else:
2742- cmd.extend([
2743- '--create-home',
2744- '--shell', shell,
2745- '--password', password,
2746- ])
2747- cmd.append(username)
2748- subprocess.check_call(cmd)
2749- user_info = pwd.getpwnam(username)
2750- return user_info
2751-
2752-
2753-def add_group(group_name, system_group=False):
2754- """Add a group to the system"""
2755- try:
2756- group_info = grp.getgrnam(group_name)
2757- log('group {0} already exists!'.format(group_name))
2758- except KeyError:
2759- log('creating group {0}'.format(group_name))
2760- cmd = ['addgroup']
2761- if system_group:
2762- cmd.append('--system')
2763- else:
2764- cmd.extend([
2765- '--group',
2766- ])
2767- cmd.append(group_name)
2768- subprocess.check_call(cmd)
2769- group_info = grp.getgrnam(group_name)
2770- return group_info
2771-
2772-
2773-def add_user_to_group(username, group):
2774- """Add a user to a group"""
2775- cmd = ['gpasswd', '-a', username, group]
2776- log("Adding user {} to group {}".format(username, group))
2777- subprocess.check_call(cmd)
2778-
2779-
2780-def rsync(from_path, to_path, flags='-r', options=None):
2781- """Replicate the contents of a path"""
2782- options = options or ['--delete', '--executability']
2783- cmd = ['/usr/bin/rsync', flags]
2784- cmd.extend(options)
2785- cmd.append(from_path)
2786- cmd.append(to_path)
2787- log(" ".join(cmd))
2788- return subprocess.check_output(cmd).decode('UTF-8').strip()
2789-
2790-
2791-def symlink(source, destination):
2792- """Create a symbolic link"""
2793- log("Symlinking {} as {}".format(source, destination))
2794- cmd = [
2795- 'ln',
2796- '-sf',
2797- source,
2798- destination,
2799- ]
2800- subprocess.check_call(cmd)
2801-
2802-
2803-def mkdir(path, owner='root', group='root', perms=0o555, force=False):
2804- """Create a directory"""
2805- log("Making dir {} {}:{} {:o}".format(path, owner, group,
2806- perms))
2807- uid = pwd.getpwnam(owner).pw_uid
2808- gid = grp.getgrnam(group).gr_gid
2809- realpath = os.path.abspath(path)
2810- path_exists = os.path.exists(realpath)
2811- if path_exists and force:
2812- if not os.path.isdir(realpath):
2813- log("Removing non-directory file {} prior to mkdir()".format(path))
2814- os.unlink(realpath)
2815- os.makedirs(realpath, perms)
2816- elif not path_exists:
2817- os.makedirs(realpath, perms)
2818- os.chown(realpath, uid, gid)
2819- os.chmod(realpath, perms)
2820-
2821-
2822-def write_file(path, content, owner='root', group='root', perms=0o444):
2823- """Create or overwrite a file with the contents of a byte string."""
2824- log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
2825- uid = pwd.getpwnam(owner).pw_uid
2826- gid = grp.getgrnam(group).gr_gid
2827- with open(path, 'wb') as target:
2828- os.fchown(target.fileno(), uid, gid)
2829- os.fchmod(target.fileno(), perms)
2830- target.write(content)
2831-
2832-
2833-def fstab_remove(mp):
2834- """Remove the given mountpoint entry from /etc/fstab
2835- """
2836- return Fstab.remove_by_mountpoint(mp)
2837-
2838-
2839-def fstab_add(dev, mp, fs, options=None):
2840- """Adds the given device entry to the /etc/fstab file
2841- """
2842- return Fstab.add(dev, mp, fs, options=options)
2843-
2844-
2845-def mount(device, mountpoint, options=None, persist=False, filesystem="ext3"):
2846- """Mount a filesystem at a particular mountpoint"""
2847- cmd_args = ['mount']
2848- if options is not None:
2849- cmd_args.extend(['-o', options])
2850- cmd_args.extend([device, mountpoint])
2851- try:
2852- subprocess.check_output(cmd_args)
2853- except subprocess.CalledProcessError as e:
2854- log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
2855- return False
2856-
2857- if persist:
2858- return fstab_add(device, mountpoint, filesystem, options=options)
2859- return True
2860-
2861-
2862-def umount(mountpoint, persist=False):
2863- """Unmount a filesystem"""
2864- cmd_args = ['umount', mountpoint]
2865- try:
2866- subprocess.check_output(cmd_args)
2867- except subprocess.CalledProcessError as e:
2868- log('Error unmounting {}\n{}'.format(mountpoint, e.output))
2869- return False
2870-
2871- if persist:
2872- return fstab_remove(mountpoint)
2873- return True
2874-
2875-
2876-def mounts():
2877- """Get a list of all mounted volumes as [[mountpoint,device],[...]]"""
2878- with open('/proc/mounts') as f:
2879- # [['/mount/point','/dev/path'],[...]]
2880- system_mounts = [m[1::-1] for m in [l.strip().split()
2881- for l in f.readlines()]]
2882- return system_mounts
2883-
2884-
2885-def file_hash(path, hash_type='md5'):
2886- """
2887- Generate a hash checksum of the contents of 'path' or None if not found.
2888-
2889- :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
2890- such as md5, sha1, sha256, sha512, etc.
2891- """
2892- if os.path.exists(path):
2893- h = getattr(hashlib, hash_type)()
2894- with open(path, 'rb') as source:
2895- h.update(source.read())
2896- return h.hexdigest()
2897- else:
2898- return None
2899-
2900-
2901-def path_hash(path):
2902- """
2903- Generate a hash checksum of all files matching 'path'. Standard wildcards
2904- like '*' and '?' are supported, see documentation for the 'glob' module for
2905- more information.
2906-
2907- :return: dict: A { filename: hash } dictionary for all matched files.
2908- Empty if none found.
2909- """
2910- return {
2911- filename: file_hash(filename)
2912- for filename in glob.iglob(path)
2913- }
2914-
2915-
2916-def check_hash(path, checksum, hash_type='md5'):
2917- """
2918- Validate a file using a cryptographic checksum.
2919-
2920- :param str checksum: Value of the checksum used to validate the file.
2921- :param str hash_type: Hash algorithm used to generate `checksum`.
2922- Can be any hash alrgorithm supported by :mod:`hashlib`,
2923- such as md5, sha1, sha256, sha512, etc.
2924- :raises ChecksumError: If the file fails the checksum
2925-
2926- """
2927- actual_checksum = file_hash(path, hash_type)
2928- if checksum != actual_checksum:
2929- raise ChecksumError("'%s' != '%s'" % (checksum, actual_checksum))
2930-
2931-
2932-class ChecksumError(ValueError):
2933- pass
2934-
2935-
2936-def restart_on_change(restart_map, stopstart=False):
2937- """Restart services based on configuration files changing
2938-
2939- This function is used a decorator, for example::
2940-
2941- @restart_on_change({
2942- '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
2943- '/etc/apache/sites-enabled/*': [ 'apache2' ]
2944- })
2945- def config_changed():
2946- pass # your code here
2947-
2948- In this example, the cinder-api and cinder-volume services
2949- would be restarted if /etc/ceph/ceph.conf is changed by the
2950- ceph_client_changed function. The apache2 service would be
2951- restarted if any file matching the pattern got changed, created
2952- or removed. Standard wildcards are supported, see documentation
2953- for the 'glob' module for more information.
2954- """
2955- def wrap(f):
2956- def wrapped_f(*args, **kwargs):
2957- checksums = {path: path_hash(path) for path in restart_map}
2958- f(*args, **kwargs)
2959- restarts = []
2960- for path in restart_map:
2961- if path_hash(path) != checksums[path]:
2962- restarts += restart_map[path]
2963- services_list = list(OrderedDict.fromkeys(restarts))
2964- if not stopstart:
2965- for service_name in services_list:
2966- service('restart', service_name)
2967- else:
2968- for action in ['stop', 'start']:
2969- for service_name in services_list:
2970- service(action, service_name)
2971- return wrapped_f
2972- return wrap
2973-
2974-
2975-def lsb_release():
2976- """Return /etc/lsb-release in a dict"""
2977- d = {}
2978- with open('/etc/lsb-release', 'r') as lsb:
2979- for l in lsb:
2980- k, v = l.split('=')
2981- d[k.strip()] = v.strip()
2982- return d
2983-
2984-
2985-def pwgen(length=None):
2986- """Generate a random pasword."""
2987- if length is None:
2988- # A random length is ok to use a weak PRNG
2989- length = random.choice(range(35, 45))
2990- alphanumeric_chars = [
2991- l for l in (string.ascii_letters + string.digits)
2992- if l not in 'l0QD1vAEIOUaeiou']
2993- # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
2994- # actual password
2995- random_generator = random.SystemRandom()
2996- random_chars = [
2997- random_generator.choice(alphanumeric_chars) for _ in range(length)]
2998- return(''.join(random_chars))
2999-
3000-
3001-def list_nics(nic_type):
3002- '''Return a list of nics of given type(s)'''
3003- if isinstance(nic_type, six.string_types):
3004- int_types = [nic_type]
3005- else:
3006- int_types = nic_type
3007- interfaces = []
3008- for int_type in int_types:
3009- cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
3010- ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
3011- ip_output = (line for line in ip_output if line)
3012- for line in ip_output:
3013- if line.split()[1].startswith(int_type):
3014- matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
3015- if matched:
3016- interface = matched.groups()[0]
3017- else:
3018- interface = line.split()[1].replace(":", "")
3019- interfaces.append(interface)
3020-
3021- return interfaces
3022-
3023-
3024-def set_nic_mtu(nic, mtu):
3025- '''Set MTU on a network interface'''
3026- cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]
3027- subprocess.check_call(cmd)
3028-
3029-
3030-def get_nic_mtu(nic):
3031- cmd = ['ip', 'addr', 'show', nic]
3032- ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
3033- mtu = ""
3034- for line in ip_output:
3035- words = line.split()
3036- if 'mtu' in words:
3037- mtu = words[words.index("mtu") + 1]
3038- return mtu
3039-
3040-
3041-def get_nic_hwaddr(nic):
3042- cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
3043- ip_output = subprocess.check_output(cmd).decode('UTF-8')
3044- hwaddr = ""
3045- words = ip_output.split()
3046- if 'link/ether' in words:
3047- hwaddr = words[words.index('link/ether') + 1]
3048- return hwaddr
3049-
3050-
3051-def cmp_pkgrevno(package, revno, pkgcache=None):
3052- '''Compare supplied revno with the revno of the installed package
3053-
3054- * 1 => Installed revno is greater than supplied arg
3055- * 0 => Installed revno is the same as supplied arg
3056- * -1 => Installed revno is less than supplied arg
3057-
3058- This function imports apt_cache function from charmhelpers.fetch if
3059- the pkgcache argument is None. Be sure to add charmhelpers.fetch if
3060- you call this function, or pass an apt_pkg.Cache() instance.
3061- '''
3062- import apt_pkg
3063- if not pkgcache:
3064- from charmhelpers.fetch import apt_cache
3065- pkgcache = apt_cache()
3066- pkg = pkgcache[package]
3067- return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
3068-
3069-
3070-@contextmanager
3071-def chdir(d):
3072- cur = os.getcwd()
3073- try:
3074- yield os.chdir(d)
3075- finally:
3076- os.chdir(cur)
3077-
3078-
3079-def chownr(path, owner, group, follow_links=True):
3080- uid = pwd.getpwnam(owner).pw_uid
3081- gid = grp.getgrnam(group).gr_gid
3082- if follow_links:
3083- chown = os.chown
3084- else:
3085- chown = os.lchown
3086-
3087- for root, dirs, files in os.walk(path):
3088- for name in dirs + files:
3089- full = os.path.join(root, name)
3090- broken_symlink = os.path.lexists(full) and not os.path.exists(full)
3091- if not broken_symlink:
3092- chown(full, uid, gid)
3093-
3094-
3095-def lchownr(path, owner, group):
3096- chownr(path, owner, group, follow_links=False)
3097-
3098-
3099-def get_total_ram():
3100- '''The total amount of system RAM in bytes.
3101-
3102- This is what is reported by the OS, and may be overcommitted when
3103- there are multiple containers hosted on the same machine.
3104- '''
3105- with open('/proc/meminfo', 'r') as f:
3106- for line in f.readlines():
3107- if line:
3108- key, value, unit = line.split()
3109- if key == 'MemTotal:':
3110- assert unit == 'kB', 'Unknown unit'
3111- return int(value) * 1024 # Classic, not KiB.
3112- raise NotImplementedError()
3113diff --git a/hooks/charmhelpers/core/services/__init__.py b/hooks/charmhelpers/core/services/__init__.py
3114deleted file mode 100644
3115index 0928158..0000000
3116--- a/hooks/charmhelpers/core/services/__init__.py
3117+++ /dev/null
3118@@ -1,18 +0,0 @@
3119-# Copyright 2014-2015 Canonical Limited.
3120-#
3121-# This file is part of charm-helpers.
3122-#
3123-# charm-helpers is free software: you can redistribute it and/or modify
3124-# it under the terms of the GNU Lesser General Public License version 3 as
3125-# published by the Free Software Foundation.
3126-#
3127-# charm-helpers is distributed in the hope that it will be useful,
3128-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3129-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3130-# GNU Lesser General Public License for more details.
3131-#
3132-# You should have received a copy of the GNU Lesser General Public License
3133-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3134-
3135-from .base import * # NOQA
3136-from .helpers import * # NOQA
3137diff --git a/hooks/charmhelpers/core/services/base.py b/hooks/charmhelpers/core/services/base.py
3138deleted file mode 100644
3139index a42660c..0000000
3140--- a/hooks/charmhelpers/core/services/base.py
3141+++ /dev/null
3142@@ -1,353 +0,0 @@
3143-# Copyright 2014-2015 Canonical Limited.
3144-#
3145-# This file is part of charm-helpers.
3146-#
3147-# charm-helpers is free software: you can redistribute it and/or modify
3148-# it under the terms of the GNU Lesser General Public License version 3 as
3149-# published by the Free Software Foundation.
3150-#
3151-# charm-helpers is distributed in the hope that it will be useful,
3152-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3153-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3154-# GNU Lesser General Public License for more details.
3155-#
3156-# You should have received a copy of the GNU Lesser General Public License
3157-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3158-
3159-import os
3160-import json
3161-from inspect import getargspec
3162-from collections import Iterable, OrderedDict
3163-
3164-from charmhelpers.core import host
3165-from charmhelpers.core import hookenv
3166-
3167-
3168-__all__ = ['ServiceManager', 'ManagerCallback',
3169- 'PortManagerCallback', 'open_ports', 'close_ports', 'manage_ports',
3170- 'service_restart', 'service_stop']
3171-
3172-
3173-class ServiceManager(object):
3174- def __init__(self, services=None):
3175- """
3176- Register a list of services, given their definitions.
3177-
3178- Service definitions are dicts in the following formats (all keys except
3179- 'service' are optional)::
3180-
3181- {
3182- "service": <service name>,
3183- "required_data": <list of required data contexts>,
3184- "provided_data": <list of provided data contexts>,
3185- "data_ready": <one or more callbacks>,
3186- "data_lost": <one or more callbacks>,
3187- "start": <one or more callbacks>,
3188- "stop": <one or more callbacks>,
3189- "ports": <list of ports to manage>,
3190- }
3191-
3192- The 'required_data' list should contain dicts of required data (or
3193- dependency managers that act like dicts and know how to collect the data).
3194- Only when all items in the 'required_data' list are populated are the list
3195- of 'data_ready' and 'start' callbacks executed. See `is_ready()` for more
3196- information.
3197-
3198- The 'provided_data' list should contain relation data providers, most likely
3199- a subclass of :class:`charmhelpers.core.services.helpers.RelationContext`,
3200- that will indicate a set of data to set on a given relation.
3201-
3202- The 'data_ready' value should be either a single callback, or a list of
3203- callbacks, to be called when all items in 'required_data' pass `is_ready()`.
3204- Each callback will be called with the service name as the only parameter.
3205- After all of the 'data_ready' callbacks are called, the 'start' callbacks
3206- are fired.
3207-
3208- The 'data_lost' value should be either a single callback, or a list of
3209- callbacks, to be called when a 'required_data' item no longer passes
3210- `is_ready()`. Each callback will be called with the service name as the
3211- only parameter. After all of the 'data_lost' callbacks are called,
3212- the 'stop' callbacks are fired.
3213-
3214- The 'start' value should be either a single callback, or a list of
3215- callbacks, to be called when starting the service, after the 'data_ready'
3216- callbacks are complete. Each callback will be called with the service
3217- name as the only parameter. This defaults to
3218- `[host.service_start, services.open_ports]`.
3219-
3220- The 'stop' value should be either a single callback, or a list of
3221- callbacks, to be called when stopping the service. If the service is
3222- being stopped because it no longer has all of its 'required_data', this
3223- will be called after all of the 'data_lost' callbacks are complete.
3224- Each callback will be called with the service name as the only parameter.
3225- This defaults to `[services.close_ports, host.service_stop]`.
3226-
3227- The 'ports' value should be a list of ports to manage. The default
3228- 'start' handler will open the ports after the service is started,
3229- and the default 'stop' handler will close the ports prior to stopping
3230- the service.
3231-
3232-
3233- Examples:
3234-
3235- The following registers an Upstart service called bingod that depends on
3236- a mongodb relation and which runs a custom `db_migrate` function prior to
3237- restarting the service, and a Runit service called spadesd::
3238-
3239- manager = services.ServiceManager([
3240- {
3241- 'service': 'bingod',
3242- 'ports': [80, 443],
3243- 'required_data': [MongoRelation(), config(), {'my': 'data'}],
3244- 'data_ready': [
3245- services.template(source='bingod.conf'),
3246- services.template(source='bingod.ini',
3247- target='/etc/bingod.ini',
3248- owner='bingo', perms=0400),
3249- ],
3250- },
3251- {
3252- 'service': 'spadesd',
3253- 'data_ready': services.template(source='spadesd_run.j2',
3254- target='/etc/sv/spadesd/run',
3255- perms=0555),
3256- 'start': runit_start,
3257- 'stop': runit_stop,
3258- },
3259- ])
3260- manager.manage()
3261- """
3262- self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
3263- self._ready = None
3264- self.services = OrderedDict()
3265- for service in services or []:
3266- service_name = service['service']
3267- self.services[service_name] = service
3268-
3269- def manage(self):
3270- """
3271- Handle the current hook by doing The Right Thing with the registered services.
3272- """
3273- hookenv._run_atstart()
3274- try:
3275- hook_name = hookenv.hook_name()
3276- if hook_name == 'stop':
3277- self.stop_services()
3278- else:
3279- self.reconfigure_services()
3280- self.provide_data()
3281- except SystemExit as x:
3282- if x.code is None or x.code == 0:
3283- hookenv._run_atexit()
3284- hookenv._run_atexit()
3285-
3286- def provide_data(self):
3287- """
3288- Set the relation data for each provider in the ``provided_data`` list.
3289-
3290- A provider must have a `name` attribute, which indicates which relation
3291- to set data on, and a `provide_data()` method, which returns a dict of
3292- data to set.
3293-
3294- The `provide_data()` method can optionally accept two parameters:
3295-
3296- * ``remote_service`` The name of the remote service that the data will
3297- be provided to. The `provide_data()` method will be called once
3298- for each connected service (not unit). This allows the method to
3299- tailor its data to the given service.
3300- * ``service_ready`` Whether or not the service definition had all of
3301- its requirements met, and thus the ``data_ready`` callbacks run.
3302-
3303- Note that the ``provided_data`` methods are now called **after** the
3304- ``data_ready`` callbacks are run. This gives the ``data_ready`` callbacks
3305- a chance to generate any data necessary for the providing to the remote
3306- services.
3307- """
3308- for service_name, service in self.services.items():
3309- service_ready = self.is_ready(service_name)
3310- for provider in service.get('provided_data', []):
3311- for relid in hookenv.relation_ids(provider.name):
3312- units = hookenv.related_units(relid)
3313- if not units:
3314- continue
3315- remote_service = units[0].split('/')[0]
3316- argspec = getargspec(provider.provide_data)
3317- if len(argspec.args) > 1:
3318- data = provider.provide_data(remote_service, service_ready)
3319- else:
3320- data = provider.provide_data()
3321- if data:
3322- hookenv.relation_set(relid, data)
3323-
3324- def reconfigure_services(self, *service_names):
3325- """
3326- Update all files for one or more registered services, and,
3327- if ready, optionally restart them.
3328-
3329- If no service names are given, reconfigures all registered services.
3330- """
3331- for service_name in service_names or self.services.keys():
3332- if self.is_ready(service_name):
3333- self.fire_event('data_ready', service_name)
3334- self.fire_event('start', service_name, default=[
3335- service_restart,
3336- manage_ports])
3337- self.save_ready(service_name)
3338- else:
3339- if self.was_ready(service_name):
3340- self.fire_event('data_lost', service_name)
3341- self.fire_event('stop', service_name, default=[
3342- manage_ports,
3343- service_stop])
3344- self.save_lost(service_name)
3345-
3346- def stop_services(self, *service_names):
3347- """
3348- Stop one or more registered services, by name.
3349-
3350- If no service names are given, stops all registered services.
3351- """
3352- for service_name in service_names or self.services.keys():
3353- self.fire_event('stop', service_name, default=[
3354- manage_ports,
3355- service_stop])
3356-
3357- def get_service(self, service_name):
3358- """
3359- Given the name of a registered service, return its service definition.
3360- """
3361- service = self.services.get(service_name)
3362- if not service:
3363- raise KeyError('Service not registered: %s' % service_name)
3364- return service
3365-
3366- def fire_event(self, event_name, service_name, default=None):
3367- """
3368- Fire a data_ready, data_lost, start, or stop event on a given service.
3369- """
3370- service = self.get_service(service_name)
3371- callbacks = service.get(event_name, default)
3372- if not callbacks:
3373- return
3374- if not isinstance(callbacks, Iterable):
3375- callbacks = [callbacks]
3376- for callback in callbacks:
3377- if isinstance(callback, ManagerCallback):
3378- callback(self, service_name, event_name)
3379- else:
3380- callback(service_name)
3381-
3382- def is_ready(self, service_name):
3383- """
3384- Determine if a registered service is ready, by checking its 'required_data'.
3385-
3386- A 'required_data' item can be any mapping type, and is considered ready
3387- if `bool(item)` evaluates as True.
3388- """
3389- service = self.get_service(service_name)
3390- reqs = service.get('required_data', [])
3391- return all(bool(req) for req in reqs)
3392-
3393- def _load_ready_file(self):
3394- if self._ready is not None:
3395- return
3396- if os.path.exists(self._ready_file):
3397- with open(self._ready_file) as fp:
3398- self._ready = set(json.load(fp))
3399- else:
3400- self._ready = set()
3401-
3402- def _save_ready_file(self):
3403- if self._ready is None:
3404- return
3405- with open(self._ready_file, 'w') as fp:
3406- json.dump(list(self._ready), fp)
3407-
3408- def save_ready(self, service_name):
3409- """
3410- Save an indicator that the given service is now data_ready.
3411- """
3412- self._load_ready_file()
3413- self._ready.add(service_name)
3414- self._save_ready_file()
3415-
3416- def save_lost(self, service_name):
3417- """
3418- Save an indicator that the given service is no longer data_ready.
3419- """
3420- self._load_ready_file()
3421- self._ready.discard(service_name)
3422- self._save_ready_file()
3423-
3424- def was_ready(self, service_name):
3425- """
3426- Determine if the given service was previously data_ready.
3427- """
3428- self._load_ready_file()
3429- return service_name in self._ready
3430-
3431-
3432-class ManagerCallback(object):
3433- """
3434- Special case of a callback that takes the `ServiceManager` instance
3435- in addition to the service name.
3436-
3437- Subclasses should implement `__call__` which should accept three parameters:
3438-
3439- * `manager` The `ServiceManager` instance
3440- * `service_name` The name of the service it's being triggered for
3441- * `event_name` The name of the event that this callback is handling
3442- """
3443- def __call__(self, manager, service_name, event_name):
3444- raise NotImplementedError()
3445-
3446-
3447-class PortManagerCallback(ManagerCallback):
3448- """
3449- Callback class that will open or close ports, for use as either
3450- a start or stop action.
3451- """
3452- def __call__(self, manager, service_name, event_name):
3453- service = manager.get_service(service_name)
3454- new_ports = service.get('ports', [])
3455- port_file = os.path.join(hookenv.charm_dir(), '.{}.ports'.format(service_name))
3456- if os.path.exists(port_file):
3457- with open(port_file) as fp:
3458- old_ports = fp.read().split(',')
3459- for old_port in old_ports:
3460- if bool(old_port):
3461- old_port = int(old_port)
3462- if old_port not in new_ports:
3463- hookenv.close_port(old_port)
3464- with open(port_file, 'w') as fp:
3465- fp.write(','.join(str(port) for port in new_ports))
3466- for port in new_ports:
3467- if event_name == 'start':
3468- hookenv.open_port(port)
3469- elif event_name == 'stop':
3470- hookenv.close_port(port)
3471-
3472-
3473-def service_stop(service_name):
3474- """
3475- Wrapper around host.service_stop to prevent spurious "unknown service"
3476- messages in the logs.
3477- """
3478- if host.service_running(service_name):
3479- host.service_stop(service_name)
3480-
3481-
3482-def service_restart(service_name):
3483- """
3484- Wrapper around host.service_restart to prevent spurious "unknown service"
3485- messages in the logs.
3486- """
3487- if host.service_available(service_name):
3488- if host.service_running(service_name):
3489- host.service_restart(service_name)
3490- else:
3491- host.service_start(service_name)
3492-
3493-
3494-# Convenience aliases
3495-open_ports = close_ports = manage_ports = PortManagerCallback()
3496diff --git a/hooks/charmhelpers/core/services/helpers.py b/hooks/charmhelpers/core/services/helpers.py
3497deleted file mode 100644
3498index 8005c41..0000000
3499--- a/hooks/charmhelpers/core/services/helpers.py
3500+++ /dev/null
3501@@ -1,267 +0,0 @@
3502-# Copyright 2014-2015 Canonical Limited.
3503-#
3504-# This file is part of charm-helpers.
3505-#
3506-# charm-helpers is free software: you can redistribute it and/or modify
3507-# it under the terms of the GNU Lesser General Public License version 3 as
3508-# published by the Free Software Foundation.
3509-#
3510-# charm-helpers is distributed in the hope that it will be useful,
3511-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3512-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3513-# GNU Lesser General Public License for more details.
3514-#
3515-# You should have received a copy of the GNU Lesser General Public License
3516-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3517-
3518-import os
3519-import yaml
3520-from charmhelpers.core import hookenv
3521-from charmhelpers.core import templating
3522-
3523-from charmhelpers.core.services.base import ManagerCallback
3524-
3525-
3526-__all__ = ['RelationContext', 'TemplateCallback',
3527- 'render_template', 'template']
3528-
3529-
3530-class RelationContext(dict):
3531- """
3532- Base class for a context generator that gets relation data from juju.
3533-
3534- Subclasses must provide the attributes `name`, which is the name of the
3535- interface of interest, `interface`, which is the type of the interface of
3536- interest, and `required_keys`, which is the set of keys required for the
3537- relation to be considered complete. The data for all interfaces matching
3538- the `name` attribute that are complete will used to populate the dictionary
3539- values (see `get_data`, below).
3540-
3541- The generated context will be namespaced under the relation :attr:`name`,
3542- to prevent potential naming conflicts.
3543-
3544- :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
3545- :param list additional_required_keys: Extend the list of :attr:`required_keys`
3546- """
3547- name = None
3548- interface = None
3549-
3550- def __init__(self, name=None, additional_required_keys=None):
3551- if not hasattr(self, 'required_keys'):
3552- self.required_keys = []
3553-
3554- if name is not None:
3555- self.name = name
3556- if additional_required_keys:
3557- self.required_keys.extend(additional_required_keys)
3558- self.get_data()
3559-
3560- def __bool__(self):
3561- """
3562- Returns True if all of the required_keys are available.
3563- """
3564- return self.is_ready()
3565-
3566- __nonzero__ = __bool__
3567-
3568- def __repr__(self):
3569- return super(RelationContext, self).__repr__()
3570-
3571- def is_ready(self):
3572- """
3573- Returns True if all of the `required_keys` are available from any units.
3574- """
3575- ready = len(self.get(self.name, [])) > 0
3576- if not ready:
3577- hookenv.log('Incomplete relation: {}'.format(self.__class__.__name__), hookenv.DEBUG)
3578- return ready
3579-
3580- def _is_ready(self, unit_data):
3581- """
3582- Helper method that tests a set of relation data and returns True if
3583- all of the `required_keys` are present.
3584- """
3585- return set(unit_data.keys()).issuperset(set(self.required_keys))
3586-
3587- def get_data(self):
3588- """
3589- Retrieve the relation data for each unit involved in a relation and,
3590- if complete, store it in a list under `self[self.name]`. This
3591- is automatically called when the RelationContext is instantiated.
3592-
3593- The units are sorted lexographically first by the service ID, then by
3594- the unit ID. Thus, if an interface has two other services, 'db:1'
3595- and 'db:2', with 'db:1' having two units, 'wordpress/0' and 'wordpress/1',
3596- and 'db:2' having one unit, 'mediawiki/0', all of which have a complete
3597- set of data, the relation data for the units will be stored in the
3598- order: 'wordpress/0', 'wordpress/1', 'mediawiki/0'.
3599-
3600- If you only care about a single unit on the relation, you can just
3601- access it as `{{ interface[0]['key'] }}`. However, if you can at all
3602- support multiple units on a relation, you should iterate over the list,
3603- like::
3604-
3605- {% for unit in interface -%}
3606- {{ unit['key'] }}{% if not loop.last %},{% endif %}
3607- {%- endfor %}
3608-
3609- Note that since all sets of relation data from all related services and
3610- units are in a single list, if you need to know which service or unit a
3611- set of data came from, you'll need to extend this class to preserve
3612- that information.
3613- """
3614- if not hookenv.relation_ids(self.name):
3615- return
3616-
3617- ns = self.setdefault(self.name, [])
3618- for rid in sorted(hookenv.relation_ids(self.name)):
3619- for unit in sorted(hookenv.related_units(rid)):
3620- reldata = hookenv.relation_get(rid=rid, unit=unit)
3621- if self._is_ready(reldata):
3622- ns.append(reldata)
3623-
3624- def provide_data(self):
3625- """
3626- Return data to be relation_set for this interface.
3627- """
3628- return {}
3629-
3630-
3631-class MysqlRelation(RelationContext):
3632- """
3633- Relation context for the `mysql` interface.
3634-
3635- :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
3636- :param list additional_required_keys: Extend the list of :attr:`required_keys`
3637- """
3638- name = 'db'
3639- interface = 'mysql'
3640-
3641- def __init__(self, *args, **kwargs):
3642- self.required_keys = ['host', 'user', 'password', 'database']
3643- RelationContext.__init__(self, *args, **kwargs)
3644-
3645-
3646-class HttpRelation(RelationContext):
3647- """
3648- Relation context for the `http` interface.
3649-
3650- :param str name: Override the relation :attr:`name`, since it can vary from charm to charm
3651- :param list additional_required_keys: Extend the list of :attr:`required_keys`
3652- """
3653- name = 'website'
3654- interface = 'http'
3655-
3656- def __init__(self, *args, **kwargs):
3657- self.required_keys = ['host', 'port']
3658- RelationContext.__init__(self, *args, **kwargs)
3659-
3660- def provide_data(self):
3661- return {
3662- 'host': hookenv.unit_get('private-address'),
3663- 'port': 80,
3664- }
3665-
3666-
3667-class RequiredConfig(dict):
3668- """
3669- Data context that loads config options with one or more mandatory options.
3670-
3671- Once the required options have been changed from their default values, all
3672- config options will be available, namespaced under `config` to prevent
3673- potential naming conflicts (for example, between a config option and a
3674- relation property).
3675-
3676- :param list *args: List of options that must be changed from their default values.
3677- """
3678-
3679- def __init__(self, *args):
3680- self.required_options = args
3681- self['config'] = hookenv.config()
3682- with open(os.path.join(hookenv.charm_dir(), 'config.yaml')) as fp:
3683- self.config = yaml.load(fp).get('options', {})
3684-
3685- def __bool__(self):
3686- for option in self.required_options:
3687- if option not in self['config']:
3688- return False
3689- current_value = self['config'][option]
3690- default_value = self.config[option].get('default')
3691- if current_value == default_value:
3692- return False
3693- if current_value in (None, '') and default_value in (None, ''):
3694- return False
3695- return True
3696-
3697- def __nonzero__(self):
3698- return self.__bool__()
3699-
3700-
3701-class StoredContext(dict):
3702- """
3703- A data context that always returns the data that it was first created with.
3704-
3705- This is useful to do a one-time generation of things like passwords, that
3706- will thereafter use the same value that was originally generated, instead
3707- of generating a new value each time it is run.
3708- """
3709- def __init__(self, file_name, config_data):
3710- """
3711- If the file exists, populate `self` with the data from the file.
3712- Otherwise, populate with the given data and persist it to the file.
3713- """
3714- if os.path.exists(file_name):
3715- self.update(self.read_context(file_name))
3716- else:
3717- self.store_context(file_name, config_data)
3718- self.update(config_data)
3719-
3720- def store_context(self, file_name, config_data):
3721- if not os.path.isabs(file_name):
3722- file_name = os.path.join(hookenv.charm_dir(), file_name)
3723- with open(file_name, 'w') as file_stream:
3724- os.fchmod(file_stream.fileno(), 0o600)
3725- yaml.dump(config_data, file_stream)
3726-
3727- def read_context(self, file_name):
3728- if not os.path.isabs(file_name):
3729- file_name = os.path.join(hookenv.charm_dir(), file_name)
3730- with open(file_name, 'r') as file_stream:
3731- data = yaml.load(file_stream)
3732- if not data:
3733- raise OSError("%s is empty" % file_name)
3734- return data
3735-
3736-
3737-class TemplateCallback(ManagerCallback):
3738- """
3739- Callback class that will render a Jinja2 template, for use as a ready
3740- action.
3741-
3742- :param str source: The template source file, relative to
3743- `$CHARM_DIR/templates`
3744- :param str target: The target to write the rendered template to
3745- :param str owner: The owner of the rendered file
3746- :param str group: The group of the rendered file
3747- :param int perms: The permissions of the rendered file
3748-
3749- """
3750- def __init__(self, source, target,
3751- owner='root', group='root', perms=0o444):
3752- self.source = source
3753- self.target = target
3754- self.owner = owner
3755- self.group = group
3756- self.perms = perms
3757-
3758- def __call__(self, manager, service_name, event_name):
3759- service = manager.get_service(service_name)
3760- context = {}
3761- for ctx in service.get('required_data', []):
3762- context.update(ctx)
3763- templating.render(self.source, self.target, context,
3764- self.owner, self.group, self.perms)
3765-
3766-
3767-# Convenience aliases for templates
3768-render_template = template = TemplateCallback
3769diff --git a/hooks/charmhelpers/core/strutils.py b/hooks/charmhelpers/core/strutils.py
3770deleted file mode 100644
3771index a2a784a..0000000
3772--- a/hooks/charmhelpers/core/strutils.py
3773+++ /dev/null
3774@@ -1,42 +0,0 @@
3775-#!/usr/bin/env python
3776-# -*- coding: utf-8 -*-
3777-
3778-# Copyright 2014-2015 Canonical Limited.
3779-#
3780-# This file is part of charm-helpers.
3781-#
3782-# charm-helpers is free software: you can redistribute it and/or modify
3783-# it under the terms of the GNU Lesser General Public License version 3 as
3784-# published by the Free Software Foundation.
3785-#
3786-# charm-helpers is distributed in the hope that it will be useful,
3787-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3788-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3789-# GNU Lesser General Public License for more details.
3790-#
3791-# You should have received a copy of the GNU Lesser General Public License
3792-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3793-
3794-import six
3795-
3796-
3797-def bool_from_string(value):
3798- """Interpret string value as boolean.
3799-
3800- Returns True if value translates to True otherwise False.
3801- """
3802- if isinstance(value, six.string_types):
3803- value = six.text_type(value)
3804- else:
3805- msg = "Unable to interpret non-string value '%s' as boolean" % (value)
3806- raise ValueError(msg)
3807-
3808- value = value.strip().lower()
3809-
3810- if value in ['y', 'yes', 'true', 't', 'on']:
3811- return True
3812- elif value in ['n', 'no', 'false', 'f', 'off']:
3813- return False
3814-
3815- msg = "Unable to interpret string value '%s' as boolean" % (value)
3816- raise ValueError(msg)
3817diff --git a/hooks/charmhelpers/core/sysctl.py b/hooks/charmhelpers/core/sysctl.py
3818deleted file mode 100644
3819index 21cc8ab..0000000
3820--- a/hooks/charmhelpers/core/sysctl.py
3821+++ /dev/null
3822@@ -1,56 +0,0 @@
3823-#!/usr/bin/env python
3824-# -*- coding: utf-8 -*-
3825-
3826-# Copyright 2014-2015 Canonical Limited.
3827-#
3828-# This file is part of charm-helpers.
3829-#
3830-# charm-helpers is free software: you can redistribute it and/or modify
3831-# it under the terms of the GNU Lesser General Public License version 3 as
3832-# published by the Free Software Foundation.
3833-#
3834-# charm-helpers is distributed in the hope that it will be useful,
3835-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3836-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3837-# GNU Lesser General Public License for more details.
3838-#
3839-# You should have received a copy of the GNU Lesser General Public License
3840-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3841-
3842-import yaml
3843-
3844-from subprocess import check_call
3845-
3846-from charmhelpers.core.hookenv import (
3847- log,
3848- DEBUG,
3849- ERROR,
3850-)
3851-
3852-__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
3853-
3854-
3855-def create(sysctl_dict, sysctl_file):
3856- """Creates a sysctl.conf file from a YAML associative array
3857-
3858- :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
3859- :type sysctl_dict: str
3860- :param sysctl_file: path to the sysctl file to be saved
3861- :type sysctl_file: str or unicode
3862- :returns: None
3863- """
3864- try:
3865- sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
3866- except yaml.YAMLError:
3867- log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
3868- level=ERROR)
3869- return
3870-
3871- with open(sysctl_file, "w") as fd:
3872- for key, value in sysctl_dict_parsed.items():
3873- fd.write("{}={}\n".format(key, value))
3874-
3875- log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
3876- level=DEBUG)
3877-
3878- check_call(["sysctl", "-p", sysctl_file])
3879diff --git a/hooks/charmhelpers/core/templating.py b/hooks/charmhelpers/core/templating.py
3880deleted file mode 100644
3881index aface26..0000000
3882--- a/hooks/charmhelpers/core/templating.py
3883+++ /dev/null
3884@@ -1,72 +0,0 @@
3885-# Copyright 2014-2015 Canonical Limited.
3886-#
3887-# This file is part of charm-helpers.
3888-#
3889-# charm-helpers is free software: you can redistribute it and/or modify
3890-# it under the terms of the GNU Lesser General Public License version 3 as
3891-# published by the Free Software Foundation.
3892-#
3893-# charm-helpers is distributed in the hope that it will be useful,
3894-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3895-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3896-# GNU Lesser General Public License for more details.
3897-#
3898-# You should have received a copy of the GNU Lesser General Public License
3899-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3900-
3901-import os
3902-
3903-from charmhelpers.core import host
3904-from charmhelpers.core import hookenv
3905-
3906-
3907-def render(source, target, context, owner='root', group='root',
3908- perms=0o444, templates_dir=None, encoding='UTF-8'):
3909- """
3910- Render a template.
3911-
3912- The `source` path, if not absolute, is relative to the `templates_dir`.
3913-
3914- The `target` path should be absolute.
3915-
3916- The context should be a dict containing the values to be replaced in the
3917- template.
3918-
3919- The `owner`, `group`, and `perms` options will be passed to `write_file`.
3920-
3921- If omitted, `templates_dir` defaults to the `templates` folder in the charm.
3922-
3923- Note: Using this requires python-jinja2; if it is not installed, calling
3924- this will attempt to use charmhelpers.fetch.apt_install to install it.
3925- """
3926- try:
3927- from jinja2 import FileSystemLoader, Environment, exceptions
3928- except ImportError:
3929- try:
3930- from charmhelpers.fetch import apt_install
3931- except ImportError:
3932- hookenv.log('Could not import jinja2, and could not import '
3933- 'charmhelpers.fetch to install it',
3934- level=hookenv.ERROR)
3935- raise
3936- apt_install('python-jinja2', fatal=True)
3937- from jinja2 import FileSystemLoader, Environment, exceptions
3938-
3939- if templates_dir is None:
3940- templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
3941- loader = Environment(loader=FileSystemLoader(templates_dir))
3942- try:
3943- source = source
3944- template = loader.get_template(source)
3945- except exceptions.TemplateNotFound as e:
3946- hookenv.log('Could not load template %s from %s.' %
3947- (source, templates_dir),
3948- level=hookenv.ERROR)
3949- raise e
3950- content = template.render(context)
3951- target_dir = os.path.dirname(target)
3952- if not os.path.exists(target_dir):
3953- # This is a terrible default directory permission, as the file
3954- # or its siblings will often contain secrets.
3955- host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
3956- host.write_file(target, content.encode(encoding), owner, group, perms)
3957diff --git a/hooks/charmhelpers/core/unitdata.py b/hooks/charmhelpers/core/unitdata.py
3958deleted file mode 100644
3959index 406a35c..0000000
3960--- a/hooks/charmhelpers/core/unitdata.py
3961+++ /dev/null
3962@@ -1,477 +0,0 @@
3963-#!/usr/bin/env python
3964-# -*- coding: utf-8 -*-
3965-#
3966-# Copyright 2014-2015 Canonical Limited.
3967-#
3968-# This file is part of charm-helpers.
3969-#
3970-# charm-helpers is free software: you can redistribute it and/or modify
3971-# it under the terms of the GNU Lesser General Public License version 3 as
3972-# published by the Free Software Foundation.
3973-#
3974-# charm-helpers is distributed in the hope that it will be useful,
3975-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3976-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3977-# GNU Lesser General Public License for more details.
3978-#
3979-# You should have received a copy of the GNU Lesser General Public License
3980-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3981-#
3982-#
3983-# Authors:
3984-# Kapil Thangavelu <kapil.foss@gmail.com>
3985-#
3986-"""
3987-Intro
3988------
3989-
3990-A simple way to store state in units. This provides a key value
3991-storage with support for versioned, transactional operation,
3992-and can calculate deltas from previous values to simplify unit logic
3993-when processing changes.
3994-
3995-
3996-Hook Integration
3997-----------------
3998-
3999-There are several extant frameworks for hook execution, including
4000-
4001- - charmhelpers.core.hookenv.Hooks
4002- - charmhelpers.core.services.ServiceManager
4003-
4004-The storage classes are framework agnostic, one simple integration is
4005-via the HookData contextmanager. It will record the current hook
4006-execution environment (including relation data, config data, etc.),
4007-setup a transaction and allow easy access to the changes from
4008-previously seen values. One consequence of the integration is the
4009-reservation of particular keys ('rels', 'unit', 'env', 'config',
4010-'charm_revisions') for their respective values.
4011-
4012-Here's a fully worked integration example using hookenv.Hooks::
4013-
4014- from charmhelper.core import hookenv, unitdata
4015-
4016- hook_data = unitdata.HookData()
4017- db = unitdata.kv()
4018- hooks = hookenv.Hooks()
4019-
4020- @hooks.hook
4021- def config_changed():
4022- # Print all changes to configuration from previously seen
4023- # values.
4024- for changed, (prev, cur) in hook_data.conf.items():
4025- print('config changed', changed,
4026- 'previous value', prev,
4027- 'current value', cur)
4028-
4029- # Get some unit specific bookeeping
4030- if not db.get('pkg_key'):
4031- key = urllib.urlopen('https://example.com/pkg_key').read()
4032- db.set('pkg_key', key)
4033-
4034- # Directly access all charm config as a mapping.
4035- conf = db.getrange('config', True)
4036-
4037- # Directly access all relation data as a mapping
4038- rels = db.getrange('rels', True)
4039-
4040- if __name__ == '__main__':
4041- with hook_data():
4042- hook.execute()
4043-
4044-
4045-A more basic integration is via the hook_scope context manager which simply
4046-manages transaction scope (and records hook name, and timestamp)::
4047-
4048- >>> from unitdata import kv
4049- >>> db = kv()
4050- >>> with db.hook_scope('install'):
4051- ... # do work, in transactional scope.
4052- ... db.set('x', 1)
4053- >>> db.get('x')
4054- 1
4055-
4056-
4057-Usage
4058------
4059-
4060-Values are automatically json de/serialized to preserve basic typing
4061-and complex data struct capabilities (dicts, lists, ints, booleans, etc).
4062-
4063-Individual values can be manipulated via get/set::
4064-
4065- >>> kv.set('y', True)
4066- >>> kv.get('y')
4067- True
4068-
4069- # We can set complex values (dicts, lists) as a single key.
4070- >>> kv.set('config', {'a': 1, 'b': True'})
4071-
4072- # Also supports returning dictionaries as a record which
4073- # provides attribute access.
4074- >>> config = kv.get('config', record=True)
4075- >>> config.b
4076- True
4077-
4078-
4079-Groups of keys can be manipulated with update/getrange::
4080-
4081- >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
4082- >>> kv.getrange('gui.', strip=True)
4083- {'z': 1, 'y': 2}
4084-
4085-When updating values, its very helpful to understand which values
4086-have actually changed and how have they changed. The storage
4087-provides a delta method to provide for this::
4088-
4089- >>> data = {'debug': True, 'option': 2}
4090- >>> delta = kv.delta(data, 'config.')
4091- >>> delta.debug.previous
4092- None
4093- >>> delta.debug.current
4094- True
4095- >>> delta
4096- {'debug': (None, True), 'option': (None, 2)}
4097-
4098-Note the delta method does not persist the actual change, it needs to
4099-be explicitly saved via 'update' method::
4100-
4101- >>> kv.update(data, 'config.')
4102-
4103-Values modified in the context of a hook scope retain historical values
4104-associated to the hookname.
4105-
4106- >>> with db.hook_scope('config-changed'):
4107- ... db.set('x', 42)
4108- >>> db.gethistory('x')
4109- [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
4110- (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
4111-
4112-"""
4113-
4114-import collections
4115-import contextlib
4116-import datetime
4117-import json
4118-import os
4119-import pprint
4120-import sqlite3
4121-import sys
4122-
4123-__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
4124-
4125-
4126-class Storage(object):
4127- """Simple key value database for local unit state within charms.
4128-
4129- Modifications are automatically committed at hook exit. That's
4130- currently regardless of exit code.
4131-
4132- To support dicts, lists, integer, floats, and booleans values
4133- are automatically json encoded/decoded.
4134- """
4135- def __init__(self, path=None):
4136- self.db_path = path
4137- if path is None:
4138- self.db_path = os.path.join(
4139- os.environ.get('CHARM_DIR', ''), '.unit-state.db')
4140- self.conn = sqlite3.connect('%s' % self.db_path)
4141- self.cursor = self.conn.cursor()
4142- self.revision = None
4143- self._closed = False
4144- self._init()
4145-
4146- def close(self):
4147- if self._closed:
4148- return
4149- self.flush(False)
4150- self.cursor.close()
4151- self.conn.close()
4152- self._closed = True
4153-
4154- def _scoped_query(self, stmt, params=None):
4155- if params is None:
4156- params = []
4157- return stmt, params
4158-
4159- def get(self, key, default=None, record=False):
4160- self.cursor.execute(
4161- *self._scoped_query(
4162- 'select data from kv where key=?', [key]))
4163- result = self.cursor.fetchone()
4164- if not result:
4165- return default
4166- if record:
4167- return Record(json.loads(result[0]))
4168- return json.loads(result[0])
4169-
4170- def getrange(self, key_prefix, strip=False):
4171- stmt = "select key, data from kv where key like '%s%%'" % key_prefix
4172- self.cursor.execute(*self._scoped_query(stmt))
4173- result = self.cursor.fetchall()
4174-
4175- if not result:
4176- return None
4177- if not strip:
4178- key_prefix = ''
4179- return dict([
4180- (k[len(key_prefix):], json.loads(v)) for k, v in result])
4181-
4182- def update(self, mapping, prefix=""):
4183- for k, v in mapping.items():
4184- self.set("%s%s" % (prefix, k), v)
4185-
4186- def unset(self, key):
4187- self.cursor.execute('delete from kv where key=?', [key])
4188- if self.revision and self.cursor.rowcount:
4189- self.cursor.execute(
4190- 'insert into kv_revisions values (?, ?, ?)',
4191- [key, self.revision, json.dumps('DELETED')])
4192-
4193- def set(self, key, value):
4194- serialized = json.dumps(value)
4195-
4196- self.cursor.execute(
4197- 'select data from kv where key=?', [key])
4198- exists = self.cursor.fetchone()
4199-
4200- # Skip mutations to the same value
4201- if exists:
4202- if exists[0] == serialized:
4203- return value
4204-
4205- if not exists:
4206- self.cursor.execute(
4207- 'insert into kv (key, data) values (?, ?)',
4208- (key, serialized))
4209- else:
4210- self.cursor.execute('''
4211- update kv
4212- set data = ?
4213- where key = ?''', [serialized, key])
4214-
4215- # Save
4216- if not self.revision:
4217- return value
4218-
4219- self.cursor.execute(
4220- 'select 1 from kv_revisions where key=? and revision=?',
4221- [key, self.revision])
4222- exists = self.cursor.fetchone()
4223-
4224- if not exists:
4225- self.cursor.execute(
4226- '''insert into kv_revisions (
4227- revision, key, data) values (?, ?, ?)''',
4228- (self.revision, key, serialized))
4229- else:
4230- self.cursor.execute(
4231- '''
4232- update kv_revisions
4233- set data = ?
4234- where key = ?
4235- and revision = ?''',
4236- [serialized, key, self.revision])
4237-
4238- return value
4239-
4240- def delta(self, mapping, prefix):
4241- """
4242- return a delta containing values that have changed.
4243- """
4244- previous = self.getrange(prefix, strip=True)
4245- if not previous:
4246- pk = set()
4247- else:
4248- pk = set(previous.keys())
4249- ck = set(mapping.keys())
4250- delta = DeltaSet()
4251-
4252- # added
4253- for k in ck.difference(pk):
4254- delta[k] = Delta(None, mapping[k])
4255-
4256- # removed
4257- for k in pk.difference(ck):
4258- delta[k] = Delta(previous[k], None)
4259-
4260- # changed
4261- for k in pk.intersection(ck):
4262- c = mapping[k]
4263- p = previous[k]
4264- if c != p:
4265- delta[k] = Delta(p, c)
4266-
4267- return delta
4268-
4269- @contextlib.contextmanager
4270- def hook_scope(self, name=""):
4271- """Scope all future interactions to the current hook execution
4272- revision."""
4273- assert not self.revision
4274- self.cursor.execute(
4275- 'insert into hooks (hook, date) values (?, ?)',
4276- (name or sys.argv[0],
4277- datetime.datetime.utcnow().isoformat()))
4278- self.revision = self.cursor.lastrowid
4279- try:
4280- yield self.revision
4281- self.revision = None
4282- except:
4283- self.flush(False)
4284- self.revision = None
4285- raise
4286- else:
4287- self.flush()
4288-
4289- def flush(self, save=True):
4290- if save:
4291- self.conn.commit()
4292- elif self._closed:
4293- return
4294- else:
4295- self.conn.rollback()
4296-
4297- def _init(self):
4298- self.cursor.execute('''
4299- create table if not exists kv (
4300- key text,
4301- data text,
4302- primary key (key)
4303- )''')
4304- self.cursor.execute('''
4305- create table if not exists kv_revisions (
4306- key text,
4307- revision integer,
4308- data text,
4309- primary key (key, revision)
4310- )''')
4311- self.cursor.execute('''
4312- create table if not exists hooks (
4313- version integer primary key autoincrement,
4314- hook text,
4315- date text
4316- )''')
4317- self.conn.commit()
4318-
4319- def gethistory(self, key, deserialize=False):
4320- self.cursor.execute(
4321- '''
4322- select kv.revision, kv.key, kv.data, h.hook, h.date
4323- from kv_revisions kv,
4324- hooks h
4325- where kv.key=?
4326- and kv.revision = h.version
4327- ''', [key])
4328- if deserialize is False:
4329- return self.cursor.fetchall()
4330- return map(_parse_history, self.cursor.fetchall())
4331-
4332- def debug(self, fh=sys.stderr):
4333- self.cursor.execute('select * from kv')
4334- pprint.pprint(self.cursor.fetchall(), stream=fh)
4335- self.cursor.execute('select * from kv_revisions')
4336- pprint.pprint(self.cursor.fetchall(), stream=fh)
4337-
4338-
4339-def _parse_history(d):
4340- return (d[0], d[1], json.loads(d[2]), d[3],
4341- datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
4342-
4343-
4344-class HookData(object):
4345- """Simple integration for existing hook exec frameworks.
4346-
4347- Records all unit information, and stores deltas for processing
4348- by the hook.
4349-
4350- Sample::
4351-
4352- from charmhelper.core import hookenv, unitdata
4353-
4354- changes = unitdata.HookData()
4355- db = unitdata.kv()
4356- hooks = hookenv.Hooks()
4357-
4358- @hooks.hook
4359- def config_changed():
4360- # View all changes to configuration
4361- for changed, (prev, cur) in changes.conf.items():
4362- print('config changed', changed,
4363- 'previous value', prev,
4364- 'current value', cur)
4365-
4366- # Get some unit specific bookeeping
4367- if not db.get('pkg_key'):
4368- key = urllib.urlopen('https://example.com/pkg_key').read()
4369- db.set('pkg_key', key)
4370-
4371- if __name__ == '__main__':
4372- with changes():
4373- hook.execute()
4374-
4375- """
4376- def __init__(self):
4377- self.kv = kv()
4378- self.conf = None
4379- self.rels = None
4380-
4381- @contextlib.contextmanager
4382- def __call__(self):
4383- from charmhelpers.core import hookenv
4384- hook_name = hookenv.hook_name()
4385-
4386- with self.kv.hook_scope(hook_name):
4387- self._record_charm_version(hookenv.charm_dir())
4388- delta_config, delta_relation = self._record_hook(hookenv)
4389- yield self.kv, delta_config, delta_relation
4390-
4391- def _record_charm_version(self, charm_dir):
4392- # Record revisions.. charm revisions are meaningless
4393- # to charm authors as they don't control the revision.
4394- # so logic dependnent on revision is not particularly
4395- # useful, however it is useful for debugging analysis.
4396- charm_rev = open(
4397- os.path.join(charm_dir, 'revision')).read().strip()
4398- charm_rev = charm_rev or '0'
4399- revs = self.kv.get('charm_revisions', [])
4400- if charm_rev not in revs:
4401- revs.append(charm_rev.strip() or '0')
4402- self.kv.set('charm_revisions', revs)
4403-
4404- def _record_hook(self, hookenv):
4405- data = hookenv.execution_environment()
4406- self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
4407- self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
4408- self.kv.set('env', dict(data['env']))
4409- self.kv.set('unit', data['unit'])
4410- self.kv.set('relid', data.get('relid'))
4411- return conf_delta, rels_delta
4412-
4413-
4414-class Record(dict):
4415-
4416- __slots__ = ()
4417-
4418- def __getattr__(self, k):
4419- if k in self:
4420- return self[k]
4421- raise AttributeError(k)
4422-
4423-
4424-class DeltaSet(Record):
4425-
4426- __slots__ = ()
4427-
4428-
4429-Delta = collections.namedtuple('Delta', ['previous', 'current'])
4430-
4431-
4432-_KV = None
4433-
4434-
4435-def kv():
4436- global _KV
4437- if _KV is None:
4438- _KV = Storage()
4439- return _KV
4440diff --git a/hooks/charmhelpers/fetch/__init__.py b/hooks/charmhelpers/fetch/__init__.py
4441deleted file mode 100644
4442index c86fa6f..0000000
4443--- a/hooks/charmhelpers/fetch/__init__.py
4444+++ /dev/null
4445@@ -1,448 +0,0 @@
4446-# Copyright 2014-2015 Canonical Limited.
4447-#
4448-# This file is part of charm-helpers.
4449-#
4450-# charm-helpers is free software: you can redistribute it and/or modify
4451-# it under the terms of the GNU Lesser General Public License version 3 as
4452-# published by the Free Software Foundation.
4453-#
4454-# charm-helpers is distributed in the hope that it will be useful,
4455-# but WITHOUT ANY WARRANTY; without even the implied warranty of
4456-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4457-# GNU Lesser General Public License for more details.
4458-#
4459-# You should have received a copy of the GNU Lesser General Public License
4460-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4461-
4462-import importlib
4463-from tempfile import NamedTemporaryFile
4464-import time
4465-from yaml import safe_load
4466-from charmhelpers.core.host import (
4467- lsb_release
4468-)
4469-import subprocess
4470-from charmhelpers.core.hookenv import (
4471- config,
4472- log,
4473-)
4474-import os
4475-
4476-import six
4477-if six.PY3:
4478- from urllib.parse import urlparse, urlunparse
4479-else:
4480- from urlparse import urlparse, urlunparse
4481-
4482-
4483-CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
4484-deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
4485-"""
4486-PROPOSED_POCKET = """# Proposed
4487-deb http://archive.ubuntu.com/ubuntu {}-proposed main universe multiverse restricted
4488-"""
4489-CLOUD_ARCHIVE_POCKETS = {
4490- # Folsom
4491- 'folsom': 'precise-updates/folsom',
4492- 'precise-folsom': 'precise-updates/folsom',
4493- 'precise-folsom/updates': 'precise-updates/folsom',
4494- 'precise-updates/folsom': 'precise-updates/folsom',
4495- 'folsom/proposed': 'precise-proposed/folsom',
4496- 'precise-folsom/proposed': 'precise-proposed/folsom',
4497- 'precise-proposed/folsom': 'precise-proposed/folsom',
4498- # Grizzly
4499- 'grizzly': 'precise-updates/grizzly',
4500- 'precise-grizzly': 'precise-updates/grizzly',
4501- 'precise-grizzly/updates': 'precise-updates/grizzly',
4502- 'precise-updates/grizzly': 'precise-updates/grizzly',
4503- 'grizzly/proposed': 'precise-proposed/grizzly',
4504- 'precise-grizzly/proposed': 'precise-proposed/grizzly',
4505- 'precise-proposed/grizzly': 'precise-proposed/grizzly',
4506- # Havana
4507- 'havana': 'precise-updates/havana',
4508- 'precise-havana': 'precise-updates/havana',
4509- 'precise-havana/updates': 'precise-updates/havana',
4510- 'precise-updates/havana': 'precise-updates/havana',
4511- 'havana/proposed': 'precise-proposed/havana',
4512- 'precise-havana/proposed': 'precise-proposed/havana',
4513- 'precise-proposed/havana': 'precise-proposed/havana',
4514- # Icehouse
4515- 'icehouse': 'precise-updates/icehouse',
4516- 'precise-icehouse': 'precise-updates/icehouse',
4517- 'precise-icehouse/updates': 'precise-updates/icehouse',
4518- 'precise-updates/icehouse': 'precise-updates/icehouse',
4519- 'icehouse/proposed': 'precise-proposed/icehouse',
4520- 'precise-icehouse/proposed': 'precise-proposed/icehouse',
4521- 'precise-proposed/icehouse': 'precise-proposed/icehouse',
4522- # Juno
4523- 'juno': 'trusty-updates/juno',
4524- 'trusty-juno': 'trusty-updates/juno',
4525- 'trusty-juno/updates': 'trusty-updates/juno',
4526- 'trusty-updates/juno': 'trusty-updates/juno',
4527- 'juno/proposed': 'trusty-proposed/juno',
4528- 'trusty-juno/proposed': 'trusty-proposed/juno',
4529- 'trusty-proposed/juno': 'trusty-proposed/juno',
4530- # Kilo
4531- 'kilo': 'trusty-updates/kilo',
4532- 'trusty-kilo': 'trusty-updates/kilo',
4533- 'trusty-kilo/updates': 'trusty-updates/kilo',
4534- 'trusty-updates/kilo': 'trusty-updates/kilo',
4535- 'kilo/proposed': 'trusty-proposed/kilo',
4536- 'trusty-kilo/proposed': 'trusty-proposed/kilo',
4537- 'trusty-proposed/kilo': 'trusty-proposed/kilo',
4538-}
4539-
4540-# The order of this list is very important. Handlers should be listed in from
4541-# least- to most-specific URL matching.
4542-FETCH_HANDLERS = (
4543- 'charmhelpers.fetch.archiveurl.ArchiveUrlFetchHandler',
4544- 'charmhelpers.fetch.bzrurl.BzrUrlFetchHandler',
4545- 'charmhelpers.fetch.giturl.GitUrlFetchHandler',
4546-)
4547-
4548-APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
4549-APT_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
4550-APT_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
4551-
4552-
4553-class SourceConfigError(Exception):
4554- pass
4555-
4556-
4557-class UnhandledSource(Exception):
4558- pass
4559-
4560-
4561-class AptLockError(Exception):
4562- pass
4563-
4564-
4565-class BaseFetchHandler(object):
4566-
4567- """Base class for FetchHandler implementations in fetch plugins"""
4568-
4569- def can_handle(self, source):
4570- """Returns True if the source can be handled. Otherwise returns
4571- a string explaining why it cannot"""
4572- return "Wrong source type"
4573-
4574- def install(self, source):
4575- """Try to download and unpack the source. Return the path to the
4576- unpacked files or raise UnhandledSource."""
4577- raise UnhandledSource("Wrong source type {}".format(source))
4578-
4579- def parse_url(self, url):
4580- return urlparse(url)
4581-
4582- def base_url(self, url):
4583- """Return url without querystring or fragment"""
4584- parts = list(self.parse_url(url))
4585- parts[4:] = ['' for i in parts[4:]]
4586- return urlunparse(parts)
4587-
4588-
4589-def filter_installed_packages(packages):
4590- """Returns a list of packages that require installation"""
4591- cache = apt_cache()
4592- _pkgs = []
4593- for package in packages:
4594- try:
4595- p = cache[package]
4596- p.current_ver or _pkgs.append(package)
4597- except KeyError:
4598- log('Package {} has no installation candidate.'.format(package),
4599- level='WARNING')
4600- _pkgs.append(package)
4601- return _pkgs
4602-
4603-
4604-def apt_cache(in_memory=True):
4605- """Build and return an apt cache"""
4606- from apt import apt_pkg
4607- apt_pkg.init()
4608- if in_memory:
4609- apt_pkg.config.set("Dir::Cache::pkgcache", "")
4610- apt_pkg.config.set("Dir::Cache::srcpkgcache", "")
4611- return apt_pkg.Cache()
4612-
4613-
4614-def apt_install(packages, options=None, fatal=False):
4615- """Install one or more packages"""
4616- if options is None:
4617- options = ['--option=Dpkg::Options::=--force-confold']
4618-
4619- cmd = ['apt-get', '--assume-yes']
4620- cmd.extend(options)
4621- cmd.append('install')
4622- if isinstance(packages, six.string_types):
4623- cmd.append(packages)
4624- else:
4625- cmd.extend(packages)
4626- log("Installing {} with options: {}".format(packages,
4627- options))
4628- _run_apt_command(cmd, fatal)
4629-
4630-
4631-def apt_upgrade(options=None, fatal=False, dist=False):
4632- """Upgrade all packages"""
4633- if options is None:
4634- options = ['--option=Dpkg::Options::=--force-confold']
4635-
4636- cmd = ['apt-get', '--assume-yes']
4637- cmd.extend(options)
4638- if dist:
4639- cmd.append('dist-upgrade')
4640- else:
4641- cmd.append('upgrade')
4642- log("Upgrading with options: {}".format(options))
4643- _run_apt_command(cmd, fatal)
4644-
4645-
4646-def apt_update(fatal=False):
4647- """Update local apt cache"""
4648- cmd = ['apt-get', 'update']
4649- _run_apt_command(cmd, fatal)
4650-
4651-
4652-def apt_purge(packages, fatal=False):
4653- """Purge one or more packages"""
4654- cmd = ['apt-get', '--assume-yes', 'purge']
4655- if isinstance(packages, six.string_types):
4656- cmd.append(packages)
4657- else:
4658- cmd.extend(packages)
4659- log("Purging {}".format(packages))
4660- _run_apt_command(cmd, fatal)
4661-
4662-
4663-def apt_mark(packages, mark, fatal=False):
4664- """Flag one or more packages using apt-mark"""
4665- log("Marking {} as {}".format(packages, mark))
4666- cmd = ['apt-mark', mark]
4667- if isinstance(packages, six.string_types):
4668- cmd.append(packages)
4669- else:
4670- cmd.extend(packages)
4671-
4672- if fatal:
4673- subprocess.check_call(cmd, universal_newlines=True)
4674- else:
4675- subprocess.call(cmd, universal_newlines=True)
4676-
4677-
4678-def apt_hold(packages, fatal=False):
4679- return apt_mark(packages, 'hold', fatal=fatal)
4680-
4681-
4682-def apt_unhold(packages, fatal=False):
4683- return apt_mark(packages, 'unhold', fatal=fatal)
4684-
4685-
4686-def add_source(source, key=None):
4687- """Add a package source to this system.
4688-
4689- @param source: a URL or sources.list entry, as supported by
4690- add-apt-repository(1). Examples::
4691-
4692- ppa:charmers/example
4693- deb https://stub:key@private.example.com/ubuntu trusty main
4694-
4695- In addition:
4696- 'proposed:' may be used to enable the standard 'proposed'
4697- pocket for the release.
4698- 'cloud:' may be used to activate official cloud archive pockets,
4699- such as 'cloud:icehouse'
4700- 'distro' may be used as a noop
4701-
4702- @param key: A key to be added to the system's APT keyring and used
4703- to verify the signatures on packages. Ideally, this should be an
4704- ASCII format GPG public key including the block headers. A GPG key
4705- id may also be used, but be aware that only insecure protocols are
4706- available to retrieve the actual public key from a public keyserver
4707- placing your Juju environment at risk. ppa and cloud archive keys
4708- are securely added automtically, so sould not be provided.
4709- """
4710- if source is None:
4711- log('Source is not present. Skipping')
4712- return
4713-
4714- if (source.startswith('ppa:') or
4715- source.startswith('http') or
4716- source.startswith('deb ') or
4717- source.startswith('cloud-archive:')):
4718- subprocess.check_call(['add-apt-repository', '--yes', source])
4719- elif source.startswith('cloud:'):
4720- apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),
4721- fatal=True)
4722- pocket = source.split(':')[-1]
4723- if pocket not in CLOUD_ARCHIVE_POCKETS:
4724- raise SourceConfigError(
4725- 'Unsupported cloud: source option %s' %
4726- pocket)
4727- actual_pocket = CLOUD_ARCHIVE_POCKETS[pocket]
4728- with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
4729- apt.write(CLOUD_ARCHIVE.format(actual_pocket))
4730- elif source == 'proposed':
4731- release = lsb_release()['DISTRIB_CODENAME']
4732- with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
4733- apt.write(PROPOSED_POCKET.format(release))
4734- elif source == 'distro':
4735- pass
4736- else:
4737- log("Unknown source: {!r}".format(source))
4738-
4739- if key:
4740- if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
4741- with NamedTemporaryFile('w+') as key_file:
4742- key_file.write(key)
4743- key_file.flush()
4744- key_file.seek(0)
4745- subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
4746- else:
4747- # Note that hkp: is in no way a secure protocol. Using a
4748- # GPG key id is pointless from a security POV unless you
4749- # absolutely trust your network and DNS.
4750- subprocess.check_call(['apt-key', 'adv', '--keyserver',
4751- 'hkp://keyserver.ubuntu.com:80', '--recv',
4752- key])
4753-
4754-
4755-def configure_sources(update=False,
4756- sources_var='install_sources',
4757- keys_var='install_keys'):
4758- """
4759- Configure multiple sources from charm configuration.
4760-
4761- The lists are encoded as yaml fragments in the configuration.
4762- The frament needs to be included as a string. Sources and their
4763- corresponding keys are of the types supported by add_source().
4764-
4765- Example config:
4766- install_sources: |
4767- - "ppa:foo"
4768- - "http://example.com/repo precise main"
4769- install_keys: |
4770- - null
4771- - "a1b2c3d4"
4772-
4773- Note that 'null' (a.k.a. None) should not be quoted.
4774- """
4775- sources = safe_load((config(sources_var) or '').strip()) or []
4776- keys = safe_load((config(keys_var) or '').strip()) or None
4777-
4778- if isinstance(sources, six.string_types):
4779- sources = [sources]
4780-
4781- if keys is None:
4782- for source in sources:
4783- add_source(source, None)
4784- else:
4785- if isinstance(keys, six.string_types):
4786- keys = [keys]
4787-
4788- if len(sources) != len(keys):
4789- raise SourceConfigError(
4790- 'Install sources and keys lists are different lengths')
4791- for source, key in zip(sources, keys):
4792- add_source(source, key)
4793- if update:
4794- apt_update(fatal=True)
4795-
4796-
4797-def install_remote(source, *args, **kwargs):
4798- """
4799- Install a file tree from a remote source
4800-
4801- The specified source should be a url of the form:
4802- scheme://[host]/path[#[option=value][&...]]
4803-
4804- Schemes supported are based on this modules submodules.
4805- Options supported are submodule-specific.
4806- Additional arguments are passed through to the submodule.
4807-
4808- For example::
4809-
4810- dest = install_remote('http://example.com/archive.tgz',
4811- checksum='deadbeef',
4812- hash_type='sha1')
4813-
4814- This will download `archive.tgz`, validate it using SHA1 and, if
4815- the file is ok, extract it and return the directory in which it
4816- was extracted. If the checksum fails, it will raise
4817- :class:`charmhelpers.core.host.ChecksumError`.
4818- """
4819- # We ONLY check for True here because can_handle may return a string
4820- # explaining why it can't handle a given source.
4821- handlers = [h for h in plugins() if h.can_handle(source) is True]
4822- installed_to = None
4823- for handler in handlers:
4824- try:
4825- installed_to = handler.install(source, *args, **kwargs)
4826- except UnhandledSource as e:
4827- log('Install source attempt unsuccessful: {}'.format(e),
4828- level='WARNING')
4829- if not installed_to:
4830- raise UnhandledSource("No handler found for source {}".format(source))
4831- return installed_to
4832-
4833-
4834-def install_from_config(config_var_name):
4835- charm_config = config()
4836- source = charm_config[config_var_name]
4837- return install_remote(source)
4838-
4839-
4840-def plugins(fetch_handlers=None):
4841- if not fetch_handlers:
4842- fetch_handlers = FETCH_HANDLERS
4843- plugin_list = []
4844- for handler_name in fetch_handlers:
4845- package, classname = handler_name.rsplit('.', 1)
4846- try:
4847- handler_class = getattr(
4848- importlib.import_module(package),
4849- classname)
4850- plugin_list.append(handler_class())
4851- except (ImportError, AttributeError):
4852- # Skip missing plugins so that they can be ommitted from
4853- # installation if desired
4854- log("FetchHandler {} not found, skipping plugin".format(
4855- handler_name))
4856- return plugin_list
4857-
4858-
4859-def _run_apt_command(cmd, fatal=False):
4860- """
4861- Run an APT command, checking output and retrying if the fatal flag is set
4862- to True.
4863-
4864- :param: cmd: str: The apt command to run.
4865- :param: fatal: bool: Whether the command's output should be checked and
4866- retried.
4867- """
4868- env = os.environ.copy()
4869-
4870- if 'DEBIAN_FRONTEND' not in env:
4871- env['DEBIAN_FRONTEND'] = 'noninteractive'
4872-
4873- if fatal:
4874- retry_count = 0
4875- result = None
4876-
4877- # If the command is considered "fatal", we need to retry if the apt
4878- # lock was not acquired.
4879-
4880- while result is None or result == APT_NO_LOCK:
4881- try:
4882- result = subprocess.check_call(cmd, env=env)
4883- except subprocess.CalledProcessError as e:
4884- retry_count = retry_count + 1
4885- if retry_count > APT_NO_LOCK_RETRY_COUNT:
4886- raise
4887- result = e.returncode
4888- log("Couldn't acquire DPKG lock. Will retry in {} seconds."
4889- "".format(APT_NO_LOCK_RETRY_DELAY))
4890- time.sleep(APT_NO_LOCK_RETRY_DELAY)
4891-
4892- else:
4893- subprocess.call(cmd, env=env)
4894diff --git a/hooks/charmhelpers/fetch/archiveurl.py b/hooks/charmhelpers/fetch/archiveurl.py
4895deleted file mode 100644
4896index efd7f9f..0000000
4897--- a/hooks/charmhelpers/fetch/archiveurl.py
4898+++ /dev/null
4899@@ -1,167 +0,0 @@
4900-# Copyright 2014-2015 Canonical Limited.
4901-#
4902-# This file is part of charm-helpers.
4903-#
4904-# charm-helpers is free software: you can redistribute it and/or modify
4905-# it under the terms of the GNU Lesser General Public License version 3 as
4906-# published by the Free Software Foundation.
4907-#
4908-# charm-helpers is distributed in the hope that it will be useful,
4909-# but WITHOUT ANY WARRANTY; without even the implied warranty of
4910-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4911-# GNU Lesser General Public License for more details.
4912-#
4913-# You should have received a copy of the GNU Lesser General Public License
4914-# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4915-
4916-import os
4917-import hashlib
4918-import re
4919-
4920-from charmhelpers.fetch import (
4921- BaseFetchHandler,
4922- UnhandledSource
4923-)
4924-from charmhelpers.payload.archive import (
4925- get_archive_handler,
4926- extract,
4927-)
4928-from charmhelpers.core.host import mkdir, check_hash
4929-
4930-import six
4931-if six.PY3:
4932- from urllib.request import (
4933- build_opener, install_opener, urlopen, urlretrieve,
4934- HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
4935- )
4936- from urllib.parse import urlparse, urlunparse, parse_qs
4937- from urllib.error import URLError
4938-else:
4939- from urllib import urlretrieve
4940- from urllib2 import (
4941- build_opener, install_opener, urlopen,
4942- HTTPPasswordMgrWithDefaultRealm, HTTPBasicAuthHandler,
4943- URLError
4944- )
4945- from urlparse import urlparse, urlunparse, parse_qs
4946-
4947-
4948-def splituser(host):
4949- '''urllib.splituser(), but six's support of this seems broken'''
4950- _userprog = re.compile('^(.*)@(.*)$')
4951- match = _userprog.match(host)
4952- if match:
4953- return match.group(1, 2)
4954- return None, host
4955-
4956-
4957-def splitpasswd(user):
4958- '''urllib.splitpasswd(), but six's support of this is missing'''
4959- _passwdprog = re.compile('^([^:]*):(.*)$', re.S)
4960- match = _passwdprog.match(user)
4961- if match:
4962- return match.group(1, 2)
4963- return user, None
4964-
4965-
4966-class ArchiveUrlFetchHandler(BaseFetchHandler):
4967- """
4968- Handler to download archive files from arbitrary URLs.
4969-
4970- Can fetch from http, https, ftp, and file URLs.
4971-
4972- Can install either tarballs (.tar, .tgz, .tbz2, etc) or zip files.
4973-
4974- Installs the contents of the archive in $CHARM_DIR/fetched/.
4975- """
4976- def can_handle(self, source):
4977- url_parts = self.parse_url(source)
4978- if url_parts.scheme not in ('http', 'https', 'ftp', 'file'):
4979- # XXX: Why is this returning a boolean and a string? It's
4980- # doomed to fail since "bool(can_handle('foo://'))" will be True.
4981- return "Wrong source type"
4982- if get_archive_handler(self.base_url(source)):
4983- return True
4984- return False
4985-
4986- def download(self, source, dest):
4987- """
4988- Download an archive file.
4989-
4990- :param str source: URL pointing to an archive file.
4991- :param str dest: Local path location to download archive file to.
4992- """
4993- # propogate all exceptions
4994- # URLError, OSError, etc
4995- proto, netloc, path, params, query, fragment = urlparse(source)
4996- if proto in ('http', 'https'):
4997- auth, barehost = splituser(netloc)
4998- if auth is not None:
4999- source = urlunparse((proto, barehost, path, params, query, fragment))
5000- username, password = splitpasswd(auth)
The diff has been truncated for viewing.

Subscribers

People subscribed via source and target branches

to all changes: