Merge lp:~hopem/charms/trusty/mysql/contrib-database-mysql into lp:charms/trusty/mysql

Proposed by Edward Hope-Morley on 2015-02-05
Status: Merged
Merged at revision: 137
Proposed branch: lp:~hopem/charms/trusty/mysql/contrib-database-mysql
Merge into: lp:charms/trusty/mysql
Diff against target: 1602 lines (+1001/-191)
24 files modified
charm-helpers.yaml (+2/-0)
hooks/charmhelpers/__init__.py (+16/-0)
hooks/charmhelpers/contrib/__init__.py (+15/-0)
hooks/charmhelpers/contrib/database/mysql.py (+372/-0)
hooks/charmhelpers/contrib/network/__init__.py (+15/-0)
hooks/charmhelpers/contrib/network/ip.py (+16/-0)
hooks/charmhelpers/contrib/peerstorage/__init__.py (+148/-0)
hooks/charmhelpers/core/__init__.py (+15/-0)
hooks/charmhelpers/core/decorators.py (+57/-0)
hooks/charmhelpers/core/fstab.py (+16/-0)
hooks/charmhelpers/core/hookenv.py (+32/-4)
hooks/charmhelpers/core/host.py (+59/-9)
hooks/charmhelpers/core/services/__init__.py (+16/-0)
hooks/charmhelpers/core/services/base.py (+16/-0)
hooks/charmhelpers/core/services/helpers.py (+16/-0)
hooks/charmhelpers/core/sysctl.py (+27/-5)
hooks/charmhelpers/core/templating.py (+19/-3)
hooks/charmhelpers/fetch/__init__.py (+24/-1)
hooks/charmhelpers/fetch/archiveurl.py (+16/-0)
hooks/charmhelpers/fetch/bzrurl.py (+25/-1)
hooks/charmhelpers/fetch/giturl.py (+26/-3)
hooks/common.py (+8/-59)
hooks/ha_relations.py (+10/-0)
hooks/shared_db_relations.py (+35/-106)
To merge this branch: bzr merge lp:~hopem/charms/trusty/mysql/contrib-database-mysql
Reviewer Review Type Date Requested Status
Liam Young 2015-02-05 Approve on 2015-02-16
Cory Johns Approve on 2015-02-13
Review Queue (community) automated testing Needs Fixing on 2015-02-11
Review via email: mp+248743@code.launchpad.net
To post a comment you must log in.

charm_unit_test #1560 mysql for hopem mp248743
    UNIT FAIL: unit-test missing

UNIT Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full unit test output: http://paste.ubuntu.com/10072413/
Build: http://10.245.162.77:8080/job/charm_unit_test/1560/

charm_lint_check #1732 mysql for hopem mp248743
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1732/

charm_amulet_test #1663 mysql for hopem mp248743
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10072425/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1663/

146. By Edward Hope-Morley on 2015-02-10

fix charm-helpers yaml

charm_lint_check #1834 mysql for hopem mp248743
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1834/

charm_unit_test #1662 mysql for hopem mp248743
    UNIT FAIL: unit-test missing

UNIT Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full unit test output: http://paste.ubuntu.com/10157219/
Build: http://10.245.162.77:8080/job/charm_unit_test/1662/

charm_amulet_test #1853 mysql for hopem mp248743
    AMULET FAIL: amulet-test missing

AMULET Results (max last 2 lines):
INFO:root:Search string not found in makefile target commands.
ERROR:root:No make target was executed.

Full amulet test output: http://paste.ubuntu.com/10157234/
Build: http://10.245.162.77:8080/job/charm_amulet_test/1853/

Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11006-results

review: Needs Fixing (automated testing)
Cory Johns (johnsca) wrote :

The automated test failure is only for Azure and seems spurious. I manually tested this relation with cs:trusty/lamp and it seems fine. +1

review: Approve
Liam Young (gnuoy) wrote :

Approve

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'charm-helpers.yaml'
2--- charm-helpers.yaml 2014-12-01 17:17:12 +0000
3+++ charm-helpers.yaml 2015-02-10 11:19:03 +0000
4@@ -5,3 +5,5 @@
5 - core
6 - fetch
7 - contrib.network.ip
8+ - contrib.database
9+ - contrib.peerstorage
10
11=== modified file 'hooks/charmhelpers/__init__.py'
12--- hooks/charmhelpers/__init__.py 2014-12-01 17:17:12 +0000
13+++ hooks/charmhelpers/__init__.py 2015-02-10 11:19:03 +0000
14@@ -1,3 +1,19 @@
15+# Copyright 2014-2015 Canonical Limited.
16+#
17+# This file is part of charm-helpers.
18+#
19+# charm-helpers is free software: you can redistribute it and/or modify
20+# it under the terms of the GNU Lesser General Public License version 3 as
21+# published by the Free Software Foundation.
22+#
23+# charm-helpers is distributed in the hope that it will be useful,
24+# but WITHOUT ANY WARRANTY; without even the implied warranty of
25+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
26+# GNU Lesser General Public License for more details.
27+#
28+# You should have received a copy of the GNU Lesser General Public License
29+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
30+
31 # Bootstrap charm-helpers, installing its dependencies if necessary using
32 # only standard libraries.
33 import subprocess
34
35=== modified file 'hooks/charmhelpers/contrib/__init__.py'
36--- hooks/charmhelpers/contrib/__init__.py 2014-05-08 10:22:43 +0000
37+++ hooks/charmhelpers/contrib/__init__.py 2015-02-10 11:19:03 +0000
38@@ -0,0 +1,15 @@
39+# Copyright 2014-2015 Canonical Limited.
40+#
41+# This file is part of charm-helpers.
42+#
43+# charm-helpers is free software: you can redistribute it and/or modify
44+# it under the terms of the GNU Lesser General Public License version 3 as
45+# published by the Free Software Foundation.
46+#
47+# charm-helpers is distributed in the hope that it will be useful,
48+# but WITHOUT ANY WARRANTY; without even the implied warranty of
49+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
50+# GNU Lesser General Public License for more details.
51+#
52+# You should have received a copy of the GNU Lesser General Public License
53+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
54
55=== added directory 'hooks/charmhelpers/contrib/database'
56=== added file 'hooks/charmhelpers/contrib/database/__init__.py'
57=== added file 'hooks/charmhelpers/contrib/database/mysql.py'
58--- hooks/charmhelpers/contrib/database/mysql.py 1970-01-01 00:00:00 +0000
59+++ hooks/charmhelpers/contrib/database/mysql.py 2015-02-10 11:19:03 +0000
60@@ -0,0 +1,372 @@
61+"""Helper for working with a MySQL database"""
62+import json
63+import socket
64+import re
65+import sys
66+import platform
67+import os
68+import glob
69+
70+from string import upper
71+
72+from charmhelpers.core.host import (
73+ mkdir,
74+ pwgen,
75+ write_file
76+)
77+from charmhelpers.core.hookenv import (
78+ relation_get,
79+ related_units,
80+ unit_get,
81+ log,
82+ DEBUG,
83+ INFO,
84+)
85+from charmhelpers.core.hookenv import config as config_get
86+from charmhelpers.fetch import (
87+ apt_install,
88+ apt_update,
89+ filter_installed_packages,
90+)
91+from charmhelpers.contrib.peerstorage import (
92+ peer_store,
93+ peer_retrieve,
94+)
95+
96+try:
97+ import MySQLdb
98+except ImportError:
99+ apt_update(fatal=True)
100+ apt_install(filter_installed_packages(['python-mysqldb']), fatal=True)
101+ import MySQLdb
102+
103+
104+class MySQLHelper(object):
105+
106+ def __init__(self, rpasswdf_template, upasswdf_template, host='localhost'):
107+ self.host = host
108+ # Password file path templates
109+ self.root_passwd_file_template = rpasswdf_template
110+ self.user_passwd_file_template = upasswdf_template
111+
112+ def connect(self, user='root', password=None):
113+ self.connection = MySQLdb.connect(user=user, host=self.host,
114+ passwd=password)
115+
116+ def database_exists(self, db_name):
117+ cursor = self.connection.cursor()
118+ try:
119+ cursor.execute("SHOW DATABASES")
120+ databases = [i[0] for i in cursor.fetchall()]
121+ finally:
122+ cursor.close()
123+
124+ return db_name in databases
125+
126+ def create_database(self, db_name):
127+ cursor = self.connection.cursor()
128+ try:
129+ cursor.execute("CREATE DATABASE {} CHARACTER SET UTF8"
130+ .format(db_name))
131+ finally:
132+ cursor.close()
133+
134+ def grant_exists(self, db_name, db_user, remote_ip):
135+ cursor = self.connection.cursor()
136+ priv_string = "GRANT ALL PRIVILEGES ON `{}`.* " \
137+ "TO '{}'@'{}'".format(db_name, db_user, remote_ip)
138+ try:
139+ cursor.execute("SHOW GRANTS for '{}'@'{}'".format(db_user,
140+ remote_ip))
141+ grants = [i[0] for i in cursor.fetchall()]
142+ except MySQLdb.OperationalError:
143+ return False
144+ finally:
145+ cursor.close()
146+
147+ # TODO: review for different grants
148+ return priv_string in grants
149+
150+ def create_grant(self, db_name, db_user, remote_ip, password):
151+ cursor = self.connection.cursor()
152+ try:
153+ # TODO: review for different grants
154+ cursor.execute("GRANT ALL PRIVILEGES ON {}.* TO '{}'@'{}' "
155+ "IDENTIFIED BY '{}'".format(db_name,
156+ db_user,
157+ remote_ip,
158+ password))
159+ finally:
160+ cursor.close()
161+
162+ def create_admin_grant(self, db_user, remote_ip, password):
163+ cursor = self.connection.cursor()
164+ try:
165+ cursor.execute("GRANT ALL PRIVILEGES ON *.* TO '{}'@'{}' "
166+ "IDENTIFIED BY '{}'".format(db_user,
167+ remote_ip,
168+ password))
169+ finally:
170+ cursor.close()
171+
172+ def cleanup_grant(self, db_user, remote_ip):
173+ cursor = self.connection.cursor()
174+ try:
175+ cursor.execute("DROP FROM mysql.user WHERE user='{}' "
176+ "AND HOST='{}'".format(db_user,
177+ remote_ip))
178+ finally:
179+ cursor.close()
180+
181+ def execute(self, sql):
182+ """Execute arbitary SQL against the database."""
183+ cursor = self.connection.cursor()
184+ try:
185+ cursor.execute(sql)
186+ finally:
187+ cursor.close()
188+
189+ def migrate_passwords_to_peer_relation(self):
190+ """Migrate any passwords storage on disk to cluster peer relation."""
191+ dirname = os.path.dirname(self.root_passwd_file_template)
192+ path = os.path.join(dirname, '*.passwd')
193+ for f in glob.glob(path):
194+ _key = os.path.basename(f)
195+ with open(f, 'r') as passwd:
196+ _value = passwd.read().strip()
197+
198+ try:
199+ peer_store(_key, _value)
200+ os.unlink(f)
201+ except ValueError:
202+ # NOTE cluster relation not yet ready - skip for now
203+ pass
204+
205+ def get_mysql_password_on_disk(self, username=None, password=None):
206+ """Retrieve, generate or store a mysql password for the provided
207+ username on disk."""
208+ if username:
209+ template = self.user_passwd_file_template
210+ passwd_file = template.format(username)
211+ else:
212+ passwd_file = self.root_passwd_file_template
213+
214+ _password = None
215+ if os.path.exists(passwd_file):
216+ with open(passwd_file, 'r') as passwd:
217+ _password = passwd.read().strip()
218+ else:
219+ mkdir(os.path.dirname(passwd_file), owner='root', group='root',
220+ perms=0o770)
221+ # Force permissions - for some reason the chmod in makedirs fails
222+ os.chmod(os.path.dirname(passwd_file), 0o770)
223+ _password = password or pwgen(length=32)
224+ write_file(passwd_file, _password, owner='root', group='root',
225+ perms=0o660)
226+
227+ return _password
228+
229+ def get_mysql_password(self, username=None, password=None):
230+ """Retrieve, generate or store a mysql password for the provided
231+ username using peer relation cluster."""
232+ self.migrate_passwords_to_peer_relation()
233+ if username:
234+ _key = 'mysql-{}.passwd'.format(username)
235+ else:
236+ _key = 'mysql.passwd'
237+
238+ try:
239+ _password = peer_retrieve(_key)
240+ if _password is None:
241+ _password = password or pwgen(length=32)
242+ peer_store(_key, _password)
243+ except ValueError:
244+ # cluster relation is not yet started; use on-disk
245+ _password = self.get_mysql_password_on_disk(username, password)
246+
247+ return _password
248+
249+ def get_mysql_root_password(self, password=None):
250+ """Retrieve or generate mysql root password for service units."""
251+ return self.get_mysql_password(username=None, password=password)
252+
253+ def get_allowed_units(self, database, username, relation_id=None):
254+ """Get list of units with access grants for database with username.
255+
256+ This is typically used to provide shared-db relations with a list of
257+ which units have been granted access to the given database.
258+ """
259+ self.connect(password=self.get_mysql_root_password())
260+ allowed_units = set()
261+ for unit in related_units(relation_id):
262+ settings = relation_get(rid=relation_id, unit=unit)
263+ # First check for setting with prefix, then without
264+ for attr in ["%s_hostname" % (database), 'hostname']:
265+ hosts = settings.get(attr, None)
266+ if hosts:
267+ break
268+
269+ if hosts:
270+ # hostname can be json-encoded list of hostnames
271+ try:
272+ hosts = json.loads(hosts)
273+ except ValueError:
274+ hosts = [hosts]
275+ else:
276+ hosts = [settings['private-address']]
277+
278+ if hosts:
279+ for host in hosts:
280+ if self.grant_exists(database, username, host):
281+ log("Grant exists for host '%s' on db '%s'" %
282+ (host, database), level=DEBUG)
283+ if unit not in allowed_units:
284+ allowed_units.add(unit)
285+ else:
286+ log("Grant does NOT exist for host '%s' on db '%s'" %
287+ (host, database), level=DEBUG)
288+ else:
289+ log("No hosts found for grant check", level=INFO)
290+
291+ return allowed_units
292+
293+ def configure_db(self, hostname, database, username, admin=False):
294+ """Configure access to database for username from hostname."""
295+ if config_get('prefer-ipv6'):
296+ remote_ip = hostname
297+ elif hostname != unit_get('private-address'):
298+ try:
299+ remote_ip = socket.gethostbyname(hostname)
300+ except Exception:
301+ # socket.gethostbyname doesn't support ipv6
302+ remote_ip = hostname
303+ else:
304+ remote_ip = '127.0.0.1'
305+
306+ self.connect(password=self.get_mysql_root_password())
307+ if not self.database_exists(database):
308+ self.create_database(database)
309+
310+ password = self.get_mysql_password(username)
311+ if not self.grant_exists(database, username, remote_ip):
312+ if not admin:
313+ self.create_grant(database, username, remote_ip, password)
314+ else:
315+ self.create_admin_grant(username, remote_ip, password)
316+
317+ return password
318+
319+
320+class PerconaClusterHelper(object):
321+
322+ # Going for the biggest page size to avoid wasted bytes. InnoDB page size is
323+ # 16MB
324+ DEFAULT_PAGE_SIZE = 16 * 1024 * 1024
325+
326+ def human_to_bytes(self, human):
327+ """Convert human readable configuration options to bytes."""
328+ num_re = re.compile('^[0-9]+$')
329+ if num_re.match(human):
330+ return human
331+
332+ factors = {
333+ 'K': 1024,
334+ 'M': 1048576,
335+ 'G': 1073741824,
336+ 'T': 1099511627776
337+ }
338+ modifier = human[-1]
339+ if modifier in factors:
340+ return int(human[:-1]) * factors[modifier]
341+
342+ if modifier == '%':
343+ total_ram = self.human_to_bytes(self.get_mem_total())
344+ if self.is_32bit_system() and total_ram > self.sys_mem_limit():
345+ total_ram = self.sys_mem_limit()
346+ factor = int(human[:-1]) * 0.01
347+ pctram = total_ram * factor
348+ return int(pctram - (pctram % self.DEFAULT_PAGE_SIZE))
349+
350+ raise ValueError("Can only convert K,M,G, or T")
351+
352+ def is_32bit_system(self):
353+ """Determine whether system is 32 or 64 bit."""
354+ try:
355+ return sys.maxsize < 2 ** 32
356+ except OverflowError:
357+ return False
358+
359+ def sys_mem_limit(self):
360+ """Determine the default memory limit for the current service unit."""
361+ if platform.machine() in ['armv7l']:
362+ _mem_limit = self.human_to_bytes('2700M') # experimentally determined
363+ else:
364+ # Limit for x86 based 32bit systems
365+ _mem_limit = self.human_to_bytes('4G')
366+
367+ return _mem_limit
368+
369+ def get_mem_total(self):
370+ """Calculate the total memory in the current service unit."""
371+ with open('/proc/meminfo') as meminfo_file:
372+ for line in meminfo_file:
373+ key, mem = line.split(':', 2)
374+ if key == 'MemTotal':
375+ mtot, modifier = mem.strip().split(' ')
376+ return '%s%s' % (mtot, upper(modifier[0]))
377+
378+ def parse_config(self):
379+ """Parse charm configuration and calculate values for config files."""
380+ config = config_get()
381+ mysql_config = {}
382+ if 'max-connections' in config:
383+ mysql_config['max_connections'] = config['max-connections']
384+
385+ # Total memory available for dataset
386+ dataset_bytes = self.human_to_bytes(config['dataset-size'])
387+ mysql_config['dataset_bytes'] = dataset_bytes
388+
389+ if 'query-cache-type' in config:
390+ # Query Cache Configuration
391+ mysql_config['query_cache_size'] = config['query-cache-size']
392+ if (config['query-cache-size'] == -1 and
393+ config['query-cache-type'] in ['ON', 'DEMAND']):
394+ # Calculate the query cache size automatically
395+ qcache_bytes = (dataset_bytes * 0.20)
396+ qcache_bytes = int(qcache_bytes -
397+ (qcache_bytes % self.DEFAULT_PAGE_SIZE))
398+ mysql_config['query_cache_size'] = qcache_bytes
399+ dataset_bytes -= qcache_bytes
400+
401+ # 5.5 allows the words, but not 5.1
402+ if config['query-cache-type'] == 'ON':
403+ mysql_config['query_cache_type'] = 1
404+ elif config['query-cache-type'] == 'DEMAND':
405+ mysql_config['query_cache_type'] = 2
406+ else:
407+ mysql_config['query_cache_type'] = 0
408+
409+ # Set a sane default key_buffer size
410+ mysql_config['key_buffer'] = self.human_to_bytes('32M')
411+
412+ if 'preferred-storage-engine' in config:
413+ # Storage engine configuration
414+ preferred_engines = config['preferred-storage-engine'].split(',')
415+ chunk_size = int(dataset_bytes / len(preferred_engines))
416+ mysql_config['innodb_flush_log_at_trx_commit'] = 1
417+ mysql_config['sync_binlog'] = 1
418+ if 'InnoDB' in preferred_engines:
419+ mysql_config['innodb_buffer_pool_size'] = chunk_size
420+ if config['tuning-level'] == 'fast':
421+ mysql_config['innodb_flush_log_at_trx_commit'] = 2
422+ else:
423+ mysql_config['innodb_buffer_pool_size'] = 0
424+
425+ mysql_config['default_storage_engine'] = preferred_engines[0]
426+ if 'MyISAM' in preferred_engines:
427+ mysql_config['key_buffer'] = chunk_size
428+
429+ if config['tuning-level'] == 'fast':
430+ mysql_config['sync_binlog'] = 0
431+
432+ return mysql_config
433
434=== modified file 'hooks/charmhelpers/contrib/network/__init__.py'
435--- hooks/charmhelpers/contrib/network/__init__.py 2014-09-21 21:48:22 +0000
436+++ hooks/charmhelpers/contrib/network/__init__.py 2015-02-10 11:19:03 +0000
437@@ -0,0 +1,15 @@
438+# Copyright 2014-2015 Canonical Limited.
439+#
440+# This file is part of charm-helpers.
441+#
442+# charm-helpers is free software: you can redistribute it and/or modify
443+# it under the terms of the GNU Lesser General Public License version 3 as
444+# published by the Free Software Foundation.
445+#
446+# charm-helpers is distributed in the hope that it will be useful,
447+# but WITHOUT ANY WARRANTY; without even the implied warranty of
448+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
449+# GNU Lesser General Public License for more details.
450+#
451+# You should have received a copy of the GNU Lesser General Public License
452+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
453
454=== modified file 'hooks/charmhelpers/contrib/network/ip.py'
455--- hooks/charmhelpers/contrib/network/ip.py 2014-11-26 12:31:33 +0000
456+++ hooks/charmhelpers/contrib/network/ip.py 2015-02-10 11:19:03 +0000
457@@ -1,3 +1,19 @@
458+# Copyright 2014-2015 Canonical Limited.
459+#
460+# This file is part of charm-helpers.
461+#
462+# charm-helpers is free software: you can redistribute it and/or modify
463+# it under the terms of the GNU Lesser General Public License version 3 as
464+# published by the Free Software Foundation.
465+#
466+# charm-helpers is distributed in the hope that it will be useful,
467+# but WITHOUT ANY WARRANTY; without even the implied warranty of
468+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
469+# GNU Lesser General Public License for more details.
470+#
471+# You should have received a copy of the GNU Lesser General Public License
472+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
473+
474 import glob
475 import re
476 import subprocess
477
478=== added directory 'hooks/charmhelpers/contrib/peerstorage'
479=== added file 'hooks/charmhelpers/contrib/peerstorage/__init__.py'
480--- hooks/charmhelpers/contrib/peerstorage/__init__.py 1970-01-01 00:00:00 +0000
481+++ hooks/charmhelpers/contrib/peerstorage/__init__.py 2015-02-10 11:19:03 +0000
482@@ -0,0 +1,148 @@
483+# Copyright 2014-2015 Canonical Limited.
484+#
485+# This file is part of charm-helpers.
486+#
487+# charm-helpers is free software: you can redistribute it and/or modify
488+# it under the terms of the GNU Lesser General Public License version 3 as
489+# published by the Free Software Foundation.
490+#
491+# charm-helpers is distributed in the hope that it will be useful,
492+# but WITHOUT ANY WARRANTY; without even the implied warranty of
493+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
494+# GNU Lesser General Public License for more details.
495+#
496+# You should have received a copy of the GNU Lesser General Public License
497+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
498+
499+import six
500+from charmhelpers.core.hookenv import relation_id as current_relation_id
501+from charmhelpers.core.hookenv import (
502+ is_relation_made,
503+ relation_ids,
504+ relation_get,
505+ local_unit,
506+ relation_set,
507+)
508+
509+
510+"""
511+This helper provides functions to support use of a peer relation
512+for basic key/value storage, with the added benefit that all storage
513+can be replicated across peer units.
514+
515+Requirement to use:
516+
517+To use this, the "peer_echo()" method has to be called form the peer
518+relation's relation-changed hook:
519+
520+@hooks.hook("cluster-relation-changed") # Adapt the to your peer relation name
521+def cluster_relation_changed():
522+ peer_echo()
523+
524+Once this is done, you can use peer storage from anywhere:
525+
526+@hooks.hook("some-hook")
527+def some_hook():
528+ # You can store and retrieve key/values this way:
529+ if is_relation_made("cluster"): # from charmhelpers.core.hookenv
530+ # There are peers available so we can work with peer storage
531+ peer_store("mykey", "myvalue")
532+ value = peer_retrieve("mykey")
533+ print value
534+ else:
535+ print "No peers joind the relation, cannot share key/values :("
536+"""
537+
538+
539+def peer_retrieve(key, relation_name='cluster'):
540+ """Retrieve a named key from peer relation `relation_name`."""
541+ cluster_rels = relation_ids(relation_name)
542+ if len(cluster_rels) > 0:
543+ cluster_rid = cluster_rels[0]
544+ return relation_get(attribute=key, rid=cluster_rid,
545+ unit=local_unit())
546+ else:
547+ raise ValueError('Unable to detect'
548+ 'peer relation {}'.format(relation_name))
549+
550+
551+def peer_retrieve_by_prefix(prefix, relation_name='cluster', delimiter='_',
552+ inc_list=None, exc_list=None):
553+ """ Retrieve k/v pairs given a prefix and filter using {inc,exc}_list """
554+ inc_list = inc_list if inc_list else []
555+ exc_list = exc_list if exc_list else []
556+ peerdb_settings = peer_retrieve('-', relation_name=relation_name)
557+ matched = {}
558+ for k, v in peerdb_settings.items():
559+ full_prefix = prefix + delimiter
560+ if k.startswith(full_prefix):
561+ new_key = k.replace(full_prefix, '')
562+ if new_key in exc_list:
563+ continue
564+ if new_key in inc_list or len(inc_list) == 0:
565+ matched[new_key] = v
566+ return matched
567+
568+
569+def peer_store(key, value, relation_name='cluster'):
570+ """Store the key/value pair on the named peer relation `relation_name`."""
571+ cluster_rels = relation_ids(relation_name)
572+ if len(cluster_rels) > 0:
573+ cluster_rid = cluster_rels[0]
574+ relation_set(relation_id=cluster_rid,
575+ relation_settings={key: value})
576+ else:
577+ raise ValueError('Unable to detect '
578+ 'peer relation {}'.format(relation_name))
579+
580+
581+def peer_echo(includes=None):
582+ """Echo filtered attributes back onto the same relation for storage.
583+
584+ This is a requirement to use the peerstorage module - it needs to be called
585+ from the peer relation's changed hook.
586+ """
587+ rdata = relation_get()
588+ echo_data = {}
589+ if includes is None:
590+ echo_data = rdata.copy()
591+ for ex in ['private-address', 'public-address']:
592+ if ex in echo_data:
593+ echo_data.pop(ex)
594+ else:
595+ for attribute, value in six.iteritems(rdata):
596+ for include in includes:
597+ if include in attribute:
598+ echo_data[attribute] = value
599+ if len(echo_data) > 0:
600+ relation_set(relation_settings=echo_data)
601+
602+
603+def peer_store_and_set(relation_id=None, peer_relation_name='cluster',
604+ peer_store_fatal=False, relation_settings=None,
605+ delimiter='_', **kwargs):
606+ """Store passed-in arguments both in argument relation and in peer storage.
607+
608+ It functions like doing relation_set() and peer_store() at the same time,
609+ with the same data.
610+
611+ @param relation_id: the id of the relation to store the data on. Defaults
612+ to the current relation.
613+ @param peer_store_fatal: Set to True, the function will raise an exception
614+ should the peer sotrage not be avialable."""
615+
616+ relation_settings = relation_settings if relation_settings else {}
617+ relation_set(relation_id=relation_id,
618+ relation_settings=relation_settings,
619+ **kwargs)
620+ if is_relation_made(peer_relation_name):
621+ for key, value in six.iteritems(dict(list(kwargs.items()) +
622+ list(relation_settings.items()))):
623+ key_prefix = relation_id or current_relation_id()
624+ peer_store(key_prefix + delimiter + key,
625+ value,
626+ relation_name=peer_relation_name)
627+ else:
628+ if peer_store_fatal:
629+ raise ValueError('Unable to detect '
630+ 'peer relation {}'.format(peer_relation_name))
631
632=== modified file 'hooks/charmhelpers/core/__init__.py'
633--- hooks/charmhelpers/core/__init__.py 2014-02-19 14:49:31 +0000
634+++ hooks/charmhelpers/core/__init__.py 2015-02-10 11:19:03 +0000
635@@ -0,0 +1,15 @@
636+# Copyright 2014-2015 Canonical Limited.
637+#
638+# This file is part of charm-helpers.
639+#
640+# charm-helpers is free software: you can redistribute it and/or modify
641+# it under the terms of the GNU Lesser General Public License version 3 as
642+# published by the Free Software Foundation.
643+#
644+# charm-helpers is distributed in the hope that it will be useful,
645+# but WITHOUT ANY WARRANTY; without even the implied warranty of
646+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
647+# GNU Lesser General Public License for more details.
648+#
649+# You should have received a copy of the GNU Lesser General Public License
650+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
651
652=== added file 'hooks/charmhelpers/core/decorators.py'
653--- hooks/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
654+++ hooks/charmhelpers/core/decorators.py 2015-02-10 11:19:03 +0000
655@@ -0,0 +1,57 @@
656+# Copyright 2014-2015 Canonical Limited.
657+#
658+# This file is part of charm-helpers.
659+#
660+# charm-helpers is free software: you can redistribute it and/or modify
661+# it under the terms of the GNU Lesser General Public License version 3 as
662+# published by the Free Software Foundation.
663+#
664+# charm-helpers is distributed in the hope that it will be useful,
665+# but WITHOUT ANY WARRANTY; without even the implied warranty of
666+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
667+# GNU Lesser General Public License for more details.
668+#
669+# You should have received a copy of the GNU Lesser General Public License
670+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
671+
672+#
673+# Copyright 2014 Canonical Ltd.
674+#
675+# Authors:
676+# Edward Hope-Morley <opentastic@gmail.com>
677+#
678+
679+import time
680+
681+from charmhelpers.core.hookenv import (
682+ log,
683+ INFO,
684+)
685+
686+
687+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
688+ """If the decorated function raises exception exc_type, allow num_retries
689+ retry attempts before raise the exception.
690+ """
691+ def _retry_on_exception_inner_1(f):
692+ def _retry_on_exception_inner_2(*args, **kwargs):
693+ retries = num_retries
694+ multiplier = 1
695+ while True:
696+ try:
697+ return f(*args, **kwargs)
698+ except exc_type:
699+ if not retries:
700+ raise
701+
702+ delay = base_delay * multiplier
703+ multiplier += 1
704+ log("Retrying '%s' %d more times (delay=%s)" %
705+ (f.__name__, retries, delay), level=INFO)
706+ retries -= 1
707+ if delay:
708+ time.sleep(delay)
709+
710+ return _retry_on_exception_inner_2
711+
712+ return _retry_on_exception_inner_1
713
714=== modified file 'hooks/charmhelpers/core/fstab.py'
715--- hooks/charmhelpers/core/fstab.py 2014-11-26 12:31:33 +0000
716+++ hooks/charmhelpers/core/fstab.py 2015-02-10 11:19:03 +0000
717@@ -1,6 +1,22 @@
718 #!/usr/bin/env python
719 # -*- coding: utf-8 -*-
720
721+# Copyright 2014-2015 Canonical Limited.
722+#
723+# This file is part of charm-helpers.
724+#
725+# charm-helpers is free software: you can redistribute it and/or modify
726+# it under the terms of the GNU Lesser General Public License version 3 as
727+# published by the Free Software Foundation.
728+#
729+# charm-helpers is distributed in the hope that it will be useful,
730+# but WITHOUT ANY WARRANTY; without even the implied warranty of
731+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
732+# GNU Lesser General Public License for more details.
733+#
734+# You should have received a copy of the GNU Lesser General Public License
735+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
736+
737 __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
738
739 import io
740
741=== modified file 'hooks/charmhelpers/core/hookenv.py'
742--- hooks/charmhelpers/core/hookenv.py 2014-11-26 12:31:33 +0000
743+++ hooks/charmhelpers/core/hookenv.py 2015-02-10 11:19:03 +0000
744@@ -1,3 +1,19 @@
745+# Copyright 2014-2015 Canonical Limited.
746+#
747+# This file is part of charm-helpers.
748+#
749+# charm-helpers is free software: you can redistribute it and/or modify
750+# it under the terms of the GNU Lesser General Public License version 3 as
751+# published by the Free Software Foundation.
752+#
753+# charm-helpers is distributed in the hope that it will be useful,
754+# but WITHOUT ANY WARRANTY; without even the implied warranty of
755+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
756+# GNU Lesser General Public License for more details.
757+#
758+# You should have received a copy of the GNU Lesser General Public License
759+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
760+
761 "Interactions with the Juju environment"
762 # Copyright 2013 Canonical Ltd.
763 #
764@@ -68,6 +84,8 @@
765 command = ['juju-log']
766 if level:
767 command += ['-l', level]
768+ if not isinstance(message, six.string_types):
769+ message = repr(message)
770 command += [message]
771 subprocess.call(command)
772
773@@ -394,21 +412,31 @@
774
775
776 @cached
777+def metadata():
778+ """Get the current charm metadata.yaml contents as a python object"""
779+ with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
780+ return yaml.safe_load(md)
781+
782+
783+@cached
784 def relation_types():
785 """Get a list of relation types supported by this charm"""
786- charmdir = os.environ.get('CHARM_DIR', '')
787- mdf = open(os.path.join(charmdir, 'metadata.yaml'))
788- md = yaml.safe_load(mdf)
789 rel_types = []
790+ md = metadata()
791 for key in ('provides', 'requires', 'peers'):
792 section = md.get(key)
793 if section:
794 rel_types.extend(section.keys())
795- mdf.close()
796 return rel_types
797
798
799 @cached
800+def charm_name():
801+ """Get the name of the current charm as is specified on metadata.yaml"""
802+ return metadata().get('name')
803+
804+
805+@cached
806 def relations():
807 """Get a nested dictionary of relation data for all related units"""
808 rels = {}
809
810=== modified file 'hooks/charmhelpers/core/host.py'
811--- hooks/charmhelpers/core/host.py 2014-11-26 12:31:33 +0000
812+++ hooks/charmhelpers/core/host.py 2015-02-10 11:19:03 +0000
813@@ -1,3 +1,19 @@
814+# Copyright 2014-2015 Canonical Limited.
815+#
816+# This file is part of charm-helpers.
817+#
818+# charm-helpers is free software: you can redistribute it and/or modify
819+# it under the terms of the GNU Lesser General Public License version 3 as
820+# published by the Free Software Foundation.
821+#
822+# charm-helpers is distributed in the hope that it will be useful,
823+# but WITHOUT ANY WARRANTY; without even the implied warranty of
824+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
825+# GNU Lesser General Public License for more details.
826+#
827+# You should have received a copy of the GNU Lesser General Public License
828+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
829+
830 """Tools for working with the host system"""
831 # Copyright 2012 Canonical Ltd.
832 #
833@@ -101,6 +117,26 @@
834 return user_info
835
836
837+def add_group(group_name, system_group=False):
838+ """Add a group to the system"""
839+ try:
840+ group_info = grp.getgrnam(group_name)
841+ log('group {0} already exists!'.format(group_name))
842+ except KeyError:
843+ log('creating group {0}'.format(group_name))
844+ cmd = ['addgroup']
845+ if system_group:
846+ cmd.append('--system')
847+ else:
848+ cmd.extend([
849+ '--group',
850+ ])
851+ cmd.append(group_name)
852+ subprocess.check_call(cmd)
853+ group_info = grp.getgrnam(group_name)
854+ return group_info
855+
856+
857 def add_user_to_group(username, group):
858 """Add a user to a group"""
859 cmd = [
860@@ -142,21 +178,24 @@
861 uid = pwd.getpwnam(owner).pw_uid
862 gid = grp.getgrnam(group).gr_gid
863 realpath = os.path.abspath(path)
864- if os.path.exists(realpath):
865- if force and not os.path.isdir(realpath):
866+ path_exists = os.path.exists(realpath)
867+ if path_exists and force:
868+ if not os.path.isdir(realpath):
869 log("Removing non-directory file {} prior to mkdir()".format(path))
870 os.unlink(realpath)
871- else:
872+ os.makedirs(realpath, perms)
873+ elif not path_exists:
874 os.makedirs(realpath, perms)
875 os.chown(realpath, uid, gid)
876+ os.chmod(realpath, perms)
877
878
879 def write_file(path, content, owner='root', group='root', perms=0o444):
880- """Create or overwrite a file with the contents of a string"""
881+ """Create or overwrite a file with the contents of a byte string."""
882 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
883 uid = pwd.getpwnam(owner).pw_uid
884 gid = grp.getgrnam(group).gr_gid
885- with open(path, 'w') as target:
886+ with open(path, 'wb') as target:
887 os.fchown(target.fileno(), uid, gid)
888 os.fchmod(target.fileno(), perms)
889 target.write(content)
890@@ -322,7 +361,7 @@
891 ip_output = (line for line in ip_output if line)
892 for line in ip_output:
893 if line.split()[1].startswith(int_type):
894- matched = re.search('.*: (bond[0-9]+\.[0-9]+)@.*', line)
895+ matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
896 if matched:
897 interface = matched.groups()[0]
898 else:
899@@ -366,10 +405,13 @@
900 * 0 => Installed revno is the same as supplied arg
901 * -1 => Installed revno is less than supplied arg
902
903+ This function imports apt_cache function from charmhelpers.fetch if
904+ the pkgcache argument is None. Be sure to add charmhelpers.fetch if
905+ you call this function, or pass an apt_pkg.Cache() instance.
906 '''
907 import apt_pkg
908- from charmhelpers.fetch import apt_cache
909 if not pkgcache:
910+ from charmhelpers.fetch import apt_cache
911 pkgcache = apt_cache()
912 pkg = pkgcache[package]
913 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
914@@ -384,13 +426,21 @@
915 os.chdir(cur)
916
917
918-def chownr(path, owner, group):
919+def chownr(path, owner, group, follow_links=True):
920 uid = pwd.getpwnam(owner).pw_uid
921 gid = grp.getgrnam(group).gr_gid
922+ if follow_links:
923+ chown = os.chown
924+ else:
925+ chown = os.lchown
926
927 for root, dirs, files in os.walk(path):
928 for name in dirs + files:
929 full = os.path.join(root, name)
930 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
931 if not broken_symlink:
932- os.chown(full, uid, gid)
933+ chown(full, uid, gid)
934+
935+
936+def lchownr(path, owner, group):
937+ chownr(path, owner, group, follow_links=False)
938
939=== modified file 'hooks/charmhelpers/core/services/__init__.py'
940--- hooks/charmhelpers/core/services/__init__.py 2014-10-22 14:38:57 +0000
941+++ hooks/charmhelpers/core/services/__init__.py 2015-02-10 11:19:03 +0000
942@@ -1,2 +1,18 @@
943+# Copyright 2014-2015 Canonical Limited.
944+#
945+# This file is part of charm-helpers.
946+#
947+# charm-helpers is free software: you can redistribute it and/or modify
948+# it under the terms of the GNU Lesser General Public License version 3 as
949+# published by the Free Software Foundation.
950+#
951+# charm-helpers is distributed in the hope that it will be useful,
952+# but WITHOUT ANY WARRANTY; without even the implied warranty of
953+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
954+# GNU Lesser General Public License for more details.
955+#
956+# You should have received a copy of the GNU Lesser General Public License
957+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
958+
959 from .base import * # NOQA
960 from .helpers import * # NOQA
961
962=== modified file 'hooks/charmhelpers/core/services/base.py'
963--- hooks/charmhelpers/core/services/base.py 2014-09-26 07:17:08 +0000
964+++ hooks/charmhelpers/core/services/base.py 2015-02-10 11:19:03 +0000
965@@ -1,3 +1,19 @@
966+# Copyright 2014-2015 Canonical Limited.
967+#
968+# This file is part of charm-helpers.
969+#
970+# charm-helpers is free software: you can redistribute it and/or modify
971+# it under the terms of the GNU Lesser General Public License version 3 as
972+# published by the Free Software Foundation.
973+#
974+# charm-helpers is distributed in the hope that it will be useful,
975+# but WITHOUT ANY WARRANTY; without even the implied warranty of
976+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
977+# GNU Lesser General Public License for more details.
978+#
979+# You should have received a copy of the GNU Lesser General Public License
980+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
981+
982 import os
983 import re
984 import json
985
986=== modified file 'hooks/charmhelpers/core/services/helpers.py'
987--- hooks/charmhelpers/core/services/helpers.py 2014-11-26 12:31:33 +0000
988+++ hooks/charmhelpers/core/services/helpers.py 2015-02-10 11:19:03 +0000
989@@ -1,3 +1,19 @@
990+# Copyright 2014-2015 Canonical Limited.
991+#
992+# This file is part of charm-helpers.
993+#
994+# charm-helpers is free software: you can redistribute it and/or modify
995+# it under the terms of the GNU Lesser General Public License version 3 as
996+# published by the Free Software Foundation.
997+#
998+# charm-helpers is distributed in the hope that it will be useful,
999+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1000+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1001+# GNU Lesser General Public License for more details.
1002+#
1003+# You should have received a copy of the GNU Lesser General Public License
1004+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1005+
1006 import os
1007 import yaml
1008 from charmhelpers.core import hookenv
1009
1010=== modified file 'hooks/charmhelpers/core/sysctl.py'
1011--- hooks/charmhelpers/core/sysctl.py 2014-10-09 10:30:08 +0000
1012+++ hooks/charmhelpers/core/sysctl.py 2015-02-10 11:19:03 +0000
1013@@ -1,6 +1,22 @@
1014 #!/usr/bin/env python
1015 # -*- coding: utf-8 -*-
1016
1017+# Copyright 2014-2015 Canonical Limited.
1018+#
1019+# This file is part of charm-helpers.
1020+#
1021+# charm-helpers is free software: you can redistribute it and/or modify
1022+# it under the terms of the GNU Lesser General Public License version 3 as
1023+# published by the Free Software Foundation.
1024+#
1025+# charm-helpers is distributed in the hope that it will be useful,
1026+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1027+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1028+# GNU Lesser General Public License for more details.
1029+#
1030+# You should have received a copy of the GNU Lesser General Public License
1031+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1032+
1033 __author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1034
1035 import yaml
1036@@ -10,25 +26,31 @@
1037 from charmhelpers.core.hookenv import (
1038 log,
1039 DEBUG,
1040+ ERROR,
1041 )
1042
1043
1044 def create(sysctl_dict, sysctl_file):
1045 """Creates a sysctl.conf file from a YAML associative array
1046
1047- :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 }
1048- :type sysctl_dict: dict
1049+ :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
1050+ :type sysctl_dict: str
1051 :param sysctl_file: path to the sysctl file to be saved
1052 :type sysctl_file: str or unicode
1053 :returns: None
1054 """
1055- sysctl_dict = yaml.load(sysctl_dict)
1056+ try:
1057+ sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
1058+ except yaml.YAMLError:
1059+ log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
1060+ level=ERROR)
1061+ return
1062
1063 with open(sysctl_file, "w") as fd:
1064- for key, value in sysctl_dict.items():
1065+ for key, value in sysctl_dict_parsed.items():
1066 fd.write("{}={}\n".format(key, value))
1067
1068- log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict),
1069+ log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
1070 level=DEBUG)
1071
1072 check_call(["sysctl", "-p", sysctl_file])
1073
1074=== modified file 'hooks/charmhelpers/core/templating.py'
1075--- hooks/charmhelpers/core/templating.py 2014-11-26 12:31:33 +0000
1076+++ hooks/charmhelpers/core/templating.py 2015-02-10 11:19:03 +0000
1077@@ -1,3 +1,19 @@
1078+# Copyright 2014-2015 Canonical Limited.
1079+#
1080+# This file is part of charm-helpers.
1081+#
1082+# charm-helpers is free software: you can redistribute it and/or modify
1083+# it under the terms of the GNU Lesser General Public License version 3 as
1084+# published by the Free Software Foundation.
1085+#
1086+# charm-helpers is distributed in the hope that it will be useful,
1087+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1088+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1089+# GNU Lesser General Public License for more details.
1090+#
1091+# You should have received a copy of the GNU Lesser General Public License
1092+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1093+
1094 import os
1095
1096 from charmhelpers.core import host
1097@@ -5,7 +21,7 @@
1098
1099
1100 def render(source, target, context, owner='root', group='root',
1101- perms=0o444, templates_dir=None):
1102+ perms=0o444, templates_dir=None, encoding='UTF-8'):
1103 """
1104 Render a template.
1105
1106@@ -48,5 +64,5 @@
1107 level=hookenv.ERROR)
1108 raise e
1109 content = template.render(context)
1110- host.mkdir(os.path.dirname(target))
1111- host.write_file(target, content, owner, group, perms)
1112+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
1113+ host.write_file(target, content.encode(encoding), owner, group, perms)
1114
1115=== modified file 'hooks/charmhelpers/fetch/__init__.py'
1116--- hooks/charmhelpers/fetch/__init__.py 2014-11-26 12:31:33 +0000
1117+++ hooks/charmhelpers/fetch/__init__.py 2015-02-10 11:19:03 +0000
1118@@ -1,3 +1,19 @@
1119+# Copyright 2014-2015 Canonical Limited.
1120+#
1121+# This file is part of charm-helpers.
1122+#
1123+# charm-helpers is free software: you can redistribute it and/or modify
1124+# it under the terms of the GNU Lesser General Public License version 3 as
1125+# published by the Free Software Foundation.
1126+#
1127+# charm-helpers is distributed in the hope that it will be useful,
1128+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1129+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1130+# GNU Lesser General Public License for more details.
1131+#
1132+# You should have received a copy of the GNU Lesser General Public License
1133+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1134+
1135 import importlib
1136 from tempfile import NamedTemporaryFile
1137 import time
1138@@ -64,9 +80,16 @@
1139 'trusty-juno/updates': 'trusty-updates/juno',
1140 'trusty-updates/juno': 'trusty-updates/juno',
1141 'juno/proposed': 'trusty-proposed/juno',
1142- 'juno/proposed': 'trusty-proposed/juno',
1143 'trusty-juno/proposed': 'trusty-proposed/juno',
1144 'trusty-proposed/juno': 'trusty-proposed/juno',
1145+ # Kilo
1146+ 'kilo': 'trusty-updates/kilo',
1147+ 'trusty-kilo': 'trusty-updates/kilo',
1148+ 'trusty-kilo/updates': 'trusty-updates/kilo',
1149+ 'trusty-updates/kilo': 'trusty-updates/kilo',
1150+ 'kilo/proposed': 'trusty-proposed/kilo',
1151+ 'trusty-kilo/proposed': 'trusty-proposed/kilo',
1152+ 'trusty-proposed/kilo': 'trusty-proposed/kilo',
1153 }
1154
1155 # The order of this list is very important. Handlers should be listed in from
1156
1157=== modified file 'hooks/charmhelpers/fetch/archiveurl.py'
1158--- hooks/charmhelpers/fetch/archiveurl.py 2014-11-26 12:31:33 +0000
1159+++ hooks/charmhelpers/fetch/archiveurl.py 2015-02-10 11:19:03 +0000
1160@@ -1,3 +1,19 @@
1161+# Copyright 2014-2015 Canonical Limited.
1162+#
1163+# This file is part of charm-helpers.
1164+#
1165+# charm-helpers is free software: you can redistribute it and/or modify
1166+# it under the terms of the GNU Lesser General Public License version 3 as
1167+# published by the Free Software Foundation.
1168+#
1169+# charm-helpers is distributed in the hope that it will be useful,
1170+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1171+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1172+# GNU Lesser General Public License for more details.
1173+#
1174+# You should have received a copy of the GNU Lesser General Public License
1175+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1176+
1177 import os
1178 import hashlib
1179 import re
1180
1181=== modified file 'hooks/charmhelpers/fetch/bzrurl.py'
1182--- hooks/charmhelpers/fetch/bzrurl.py 2014-11-26 12:31:33 +0000
1183+++ hooks/charmhelpers/fetch/bzrurl.py 2015-02-10 11:19:03 +0000
1184@@ -1,3 +1,19 @@
1185+# Copyright 2014-2015 Canonical Limited.
1186+#
1187+# This file is part of charm-helpers.
1188+#
1189+# charm-helpers is free software: you can redistribute it and/or modify
1190+# it under the terms of the GNU Lesser General Public License version 3 as
1191+# published by the Free Software Foundation.
1192+#
1193+# charm-helpers is distributed in the hope that it will be useful,
1194+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1195+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1196+# GNU Lesser General Public License for more details.
1197+#
1198+# You should have received a copy of the GNU Lesser General Public License
1199+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1200+
1201 import os
1202 from charmhelpers.fetch import (
1203 BaseFetchHandler,
1204@@ -11,10 +27,12 @@
1205
1206 try:
1207 from bzrlib.branch import Branch
1208+ from bzrlib import bzrdir, workingtree, errors
1209 except ImportError:
1210 from charmhelpers.fetch import apt_install
1211 apt_install("python-bzrlib")
1212 from bzrlib.branch import Branch
1213+ from bzrlib import bzrdir, workingtree, errors
1214
1215
1216 class BzrUrlFetchHandler(BaseFetchHandler):
1217@@ -35,8 +53,14 @@
1218 from bzrlib.plugin import load_plugins
1219 load_plugins()
1220 try:
1221+ local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
1222+ except errors.AlreadyControlDirError:
1223+ local_branch = Branch.open(dest)
1224+ try:
1225 remote_branch = Branch.open(source)
1226- remote_branch.bzrdir.sprout(dest).open_branch()
1227+ remote_branch.push(local_branch)
1228+ tree = workingtree.WorkingTree.open(dest)
1229+ tree.update()
1230 except Exception as e:
1231 raise e
1232
1233
1234=== modified file 'hooks/charmhelpers/fetch/giturl.py'
1235--- hooks/charmhelpers/fetch/giturl.py 2014-11-26 12:31:33 +0000
1236+++ hooks/charmhelpers/fetch/giturl.py 2015-02-10 11:19:03 +0000
1237@@ -1,3 +1,19 @@
1238+# Copyright 2014-2015 Canonical Limited.
1239+#
1240+# This file is part of charm-helpers.
1241+#
1242+# charm-helpers is free software: you can redistribute it and/or modify
1243+# it under the terms of the GNU Lesser General Public License version 3 as
1244+# published by the Free Software Foundation.
1245+#
1246+# charm-helpers is distributed in the hope that it will be useful,
1247+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1248+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1249+# GNU Lesser General Public License for more details.
1250+#
1251+# You should have received a copy of the GNU Lesser General Public License
1252+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1253+
1254 import os
1255 from charmhelpers.fetch import (
1256 BaseFetchHandler,
1257@@ -16,6 +32,8 @@
1258 apt_install("python-git")
1259 from git import Repo
1260
1261+from git.exc import GitCommandError
1262+
1263
1264 class GitUrlFetchHandler(BaseFetchHandler):
1265 """Handler for git branches via generic and github URLs"""
1266@@ -34,15 +52,20 @@
1267 repo = Repo.clone_from(source, dest)
1268 repo.git.checkout(branch)
1269
1270- def install(self, source, branch="master"):
1271+ def install(self, source, branch="master", dest=None):
1272 url_parts = self.parse_url(source)
1273 branch_name = url_parts.path.strip("/").split("/")[-1]
1274- dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
1275- branch_name)
1276+ if dest:
1277+ dest_dir = os.path.join(dest, branch_name)
1278+ else:
1279+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
1280+ branch_name)
1281 if not os.path.exists(dest_dir):
1282 mkdir(dest_dir, perms=0o755)
1283 try:
1284 self.clone(source, dest_dir, branch)
1285+ except GitCommandError as e:
1286+ raise UnhandledSource(e.message)
1287 except OSError as e:
1288 raise UnhandledSource(e.strerror)
1289 return dest_dir
1290
1291=== modified file 'hooks/common.py'
1292--- hooks/common.py 2015-01-07 14:18:57 +0000
1293+++ hooks/common.py 2015-02-10 11:19:03 +0000
1294@@ -6,6 +6,7 @@
1295 import shutil
1296 from charmhelpers.core import hookenv, host
1297 from charmhelpers.core.templating import render
1298+from charmhelpers.contrib.database.mysql import MySQLHelper
1299
1300
1301 def get_service_user_file(service):
1302@@ -60,71 +61,19 @@
1303 broken = os.path.exists(broken_path)
1304
1305
1306+def get_db_helper():
1307+ return MySQLHelper(rpasswdf_template='/var/lib/mysql/mysql.passwd',
1308+ upasswdf_template='/var/lib/mysql/mysql-{}.passwd')
1309+
1310+
1311 def get_db_cursor():
1312 # Connect to mysql
1313- passwd = open("/var/lib/mysql/mysql.passwd").read().strip()
1314+ db_helper = get_db_helper()
1315+ passwd = db_helper.get_mysql_root_password()
1316 connection = MySQLdb.connect(user="root", host="localhost", passwd=passwd)
1317 return connection.cursor()
1318
1319
1320-def database_exists(db_name):
1321- cursor = get_db_cursor()
1322- try:
1323- cursor.execute("SHOW DATABASES")
1324- databases = [i[0] for i in cursor.fetchall()]
1325- finally:
1326- cursor.close()
1327- return db_name in databases
1328-
1329-
1330-def create_database(db_name):
1331- cursor = get_db_cursor()
1332- try:
1333- cursor.execute("CREATE DATABASE {}".format(db_name))
1334- finally:
1335- cursor.close()
1336-
1337-
1338-def grant_exists(db_name, db_user, remote_ip):
1339- cursor = get_db_cursor()
1340- priv_string = "GRANT ALL PRIVILEGES ON `{}`.* " \
1341- "TO '{}'@'{}'".format(db_name, db_user, remote_ip)
1342- try:
1343- cursor.execute("SHOW GRANTS for '{}'@'{}'".format(db_user,
1344- remote_ip))
1345- grants = [i[0] for i in cursor.fetchall()]
1346- except MySQLdb.OperationalError:
1347- print "No grants found"
1348- return False
1349- finally:
1350- cursor.close()
1351- return priv_string in grants
1352-
1353-
1354-def create_grant(db_name, db_user,
1355- remote_ip, password):
1356- cursor = get_db_cursor()
1357- try:
1358- cursor.execute("GRANT ALL PRIVILEGES ON {}.* TO '{}'@'{}' "
1359- "IDENTIFIED BY '{}'".format(db_name,
1360- db_user,
1361- remote_ip,
1362- password))
1363- finally:
1364- cursor.close()
1365-
1366-
1367-def cleanup_grant(db_user,
1368- remote_ip):
1369- cursor = get_db_cursor()
1370- try:
1371- cursor.execute("DROP FROM mysql.user WHERE user='{}' "
1372- "AND HOST='{}'".format(db_user,
1373- remote_ip))
1374- finally:
1375- cursor.close()
1376-
1377-
1378 def migrate_to_mount(new_path):
1379 """Invoked when new mountpoint appears. This function safely migrates
1380 MySQL data from local disk to persistent storage (only if needed)
1381
1382=== modified file 'hooks/ha_relations.py'
1383--- hooks/ha_relations.py 2014-09-21 22:03:41 +0000
1384+++ hooks/ha_relations.py 2015-02-10 11:19:03 +0000
1385@@ -7,6 +7,10 @@
1386 import lib.ceph_utils as ceph
1387 import lib.cluster_utils as cluster
1388
1389+from charmhelpers.contrib.peerstorage import (
1390+ peer_echo,
1391+)
1392+
1393 # CEPH
1394 DATA_SRC_DST = '/var/lib/mysql'
1395 SERVICE_NAME = os.getenv('JUJU_UNIT_NAME').split('/')[0]
1396@@ -14,6 +18,11 @@
1397 LEADER_RES = 'res_mysql_vip'
1398
1399
1400+def cluster_changed():
1401+ # Echo any passwords placed on peer relation
1402+ peer_echo(includes=['.passwd'])
1403+
1404+
1405 def ha_relation_joined():
1406 vip = utils.config_get('vip')
1407 vip_iface = utils.config_get('vip_iface')
1408@@ -148,6 +157,7 @@
1409 "ha-relation-changed": ha_relation_changed,
1410 "ceph-relation-joined": ceph_joined,
1411 "ceph-relation-changed": ceph_changed,
1412+ "cluster-relation-changed": cluster_changed,
1413 }
1414
1415 utils.do_hooks(hooks)
1416
1417=== modified file 'hooks/shared_db_relations.py'
1418--- hooks/shared_db_relations.py 2014-11-26 16:35:19 +0000
1419+++ hooks/shared_db_relations.py 2015-02-10 11:19:03 +0000
1420@@ -7,18 +7,12 @@
1421 #
1422 # Author: Adam Gandelman <adam.gandelman@canonical.com>
1423
1424-
1425-from common import (
1426- database_exists,
1427- create_database,
1428- grant_exists,
1429- create_grant)
1430 import subprocess
1431 import json
1432-import socket
1433-import os
1434 import lib.utils as utils
1435 import lib.cluster_utils as cluster
1436+
1437+from common import get_db_helper
1438 from charmhelpers.core import hookenv
1439 from charmhelpers.contrib.network.ip import (
1440 get_ipv6_addr
1441@@ -50,71 +44,6 @@
1442
1443
1444 def shared_db_changed():
1445-
1446- def get_allowed_units(database, username):
1447- allowed_units = set()
1448- for relid in hookenv.relation_ids('shared-db'):
1449- for unit in hookenv.related_units(relid):
1450- attr = "%s_%s" % (database, 'hostname')
1451- hosts = hookenv.relation_get(attribute=attr, unit=unit,
1452- rid=relid)
1453- if not hosts:
1454- hosts = [hookenv.relation_get(attribute='private-address',
1455- unit=unit, rid=relid)]
1456- else:
1457- # hostname can be json-encoded list of hostnames
1458- try:
1459- hosts = json.loads(hosts)
1460- except ValueError:
1461- pass
1462-
1463- if not isinstance(hosts, list):
1464- hosts = [hosts]
1465-
1466- if hosts:
1467- for host in hosts:
1468- utils.juju_log('INFO', "Checking host '%s' grant" %
1469- (host))
1470- if grant_exists(database, username, host):
1471- if unit not in allowed_units:
1472- allowed_units.add(unit)
1473- else:
1474- utils.juju_log('INFO', "No hosts found for grant check")
1475-
1476- return allowed_units
1477-
1478- def configure_db(hostname,
1479- database,
1480- username):
1481- passwd_file = "/var/lib/mysql/mysql-{}.passwd".format(username)
1482- if hostname != local_hostname:
1483- try:
1484- remote_ip = socket.gethostbyname(hostname)
1485- except Exception:
1486- # socket.gethostbyname doesn't support ipv6
1487- remote_ip = hostname
1488- else:
1489- remote_ip = '127.0.0.1'
1490-
1491- if not os.path.exists(passwd_file):
1492- password = pwgen()
1493- with open(passwd_file, 'w') as pfile:
1494- pfile.write(password)
1495- os.chmod(pfile.name, 0600)
1496- else:
1497- with open(passwd_file) as pfile:
1498- password = pfile.read().strip()
1499-
1500- if not database_exists(database):
1501- create_database(database)
1502- if not grant_exists(database,
1503- username,
1504- remote_ip):
1505- create_grant(database,
1506- username,
1507- remote_ip, password)
1508- return password
1509-
1510 if not cluster.eligible_leader(LEADER_RES):
1511 utils.juju_log('INFO',
1512 'MySQL service is peered, bailing shared-db relation'
1513@@ -132,6 +61,8 @@
1514 'username',
1515 'hostname'])
1516
1517+ db_helper = get_db_helper()
1518+
1519 if singleset.issubset(settings):
1520 # Process a single database configuration
1521 hostname = settings['hostname']
1522@@ -142,26 +73,23 @@
1523 try:
1524 hostname = json.loads(hostname)
1525 except ValueError:
1526- pass
1527-
1528- if isinstance(hostname, list):
1529- for host in hostname:
1530- password = configure_db(host, database, username)
1531- else:
1532- password = configure_db(hostname, database, username)
1533-
1534- allowed_units = " ".join(unit_sorted(get_allowed_units(database,
1535- username)))
1536-
1537- if not cluster.is_clustered():
1538- utils.relation_set(db_host=local_hostname,
1539- password=password,
1540- allowed_units=allowed_units)
1541- else:
1542- utils.relation_set(db_host=utils.config_get("vip"),
1543- password=password,
1544- allowed_units=allowed_units)
1545-
1546+ hostname = [hostname]
1547+
1548+ for host in hostname:
1549+ password = db_helper.configure_db(host, database, username)
1550+
1551+ allowed_units = db_helper.get_allowed_units(database, username)
1552+ allowed_units = unit_sorted(allowed_units)
1553+ allowed_units = ' '.join(allowed_units)
1554+
1555+ if cluster.is_clustered():
1556+ db_host = utils.config_get("vip")
1557+ else:
1558+ db_host = local_hostname
1559+
1560+ utils.relation_set(db_host=db_host,
1561+ password=password,
1562+ allowed_units=allowed_units)
1563 else:
1564 # Process multiple database setup requests.
1565 # from incoming relation data:
1566@@ -195,22 +123,23 @@
1567 database = databases[db]['database']
1568 hostname = databases[db]['hostname']
1569 username = databases[db]['username']
1570+
1571 try:
1572+ # Can be json-encoded list of hostnames
1573 hostname = json.loads(hostname)
1574 except ValueError:
1575- hostname = hostname
1576-
1577- if isinstance(hostname, list):
1578- for host in hostname:
1579- password = configure_db(host, database, username)
1580- else:
1581- password = configure_db(hostname, database, username)
1582-
1583- return_data['_'.join([db, 'password'])] = password
1584- allowed_units = unit_sorted(get_allowed_units(database,
1585- username))
1586- return_data['_'.join([db, 'allowed_units'])] = \
1587- " ".join(allowed_units)
1588+ # Otherwise expected to be single hostname
1589+ hostname = [hostname]
1590+
1591+ for host in hostname:
1592+ password = db_helper.configure_db(host, database, username)
1593+
1594+ a_units = db_helper.get_allowed_units(database, username)
1595+ a_units = ' '.join(unit_sorted(a_units))
1596+ return_data['%s_allowed_units' % (db)] = a_units
1597+
1598+ return_data['%s_password' % (db)] = password
1599+
1600 if len(return_data) > 0:
1601 utils.relation_set(**return_data)
1602 if not cluster.is_clustered():

Subscribers

People subscribed via source and target branches

to all changes: