Merge lp:~petevg/charms/trusty/mariadb/fixes-for-xenial into lp:~dbart/charms/trusty/mariadb/trunk

Proposed by Pete Vander Giessen on 2016-07-12
Status: Needs review
Proposed branch: lp:~petevg/charms/trusty/mariadb/fixes-for-xenial
Merge into: lp:~dbart/charms/trusty/mariadb/trunk
Diff against target: 3910 lines (+2438/-419)
33 files modified
README.md (+39/-50)
config.yaml (+10/-35)
hooks/common.py (+2/-2)
hooks/config-changed (+25/-46)
hooks/install (+2/-2)
hooks/start (+2/-1)
lib/charmhelpers/__init__.py (+38/-0)
lib/charmhelpers/core/__init__.py (+15/-0)
lib/charmhelpers/core/decorators.py (+57/-0)
lib/charmhelpers/core/files.py (+45/-0)
lib/charmhelpers/core/fstab.py (+19/-3)
lib/charmhelpers/core/hookenv.py (+516/-47)
lib/charmhelpers/core/host.py (+442/-74)
lib/charmhelpers/core/hugepage.py (+71/-0)
lib/charmhelpers/core/kernel.py (+68/-0)
lib/charmhelpers/core/services/__init__.py (+16/-0)
lib/charmhelpers/core/services/base.py (+59/-19)
lib/charmhelpers/core/services/helpers.py (+59/-10)
lib/charmhelpers/core/strutils.py (+72/-0)
lib/charmhelpers/core/sysctl.py (+28/-6)
lib/charmhelpers/core/templating.py (+37/-8)
lib/charmhelpers/core/unitdata.py (+521/-0)
lib/charmhelpers/fetch/__init__.py (+75/-22)
lib/charmhelpers/fetch/archiveurl.py (+34/-12)
lib/charmhelpers/fetch/bzrurl.py (+51/-28)
lib/charmhelpers/fetch/giturl.py (+45/-23)
lib/charmhelpers/payload/__init__.py (+16/-0)
lib/charmhelpers/payload/archive.py (+16/-0)
lib/charmhelpers/payload/execd.py (+16/-0)
metadata.yaml (+1/-0)
revision (+1/-1)
scripts/charm_helpers_sync.py (+38/-8)
tests/10-deploy-and-upgrade (+2/-22)
To merge this branch: bzr merge lp:~petevg/charms/trusty/mariadb/fixes-for-xenial
Reviewer Review Type Date Requested Status
Daniel Bartholomew 2016-07-12 Pending
Review via email: mp+299826@code.launchpad.net

Description of the change

Hi Daniel,

As part of a project to ship IBM a charm that works on their z Linux distro, I made a version of the mariadb charm that works on xenial.

The changes are:

1) Ported config-changed and install to python3 (python2 is not installed by default on xenial, though it does get installed when we install python-mysql in the config-changed script, so install and config-changed are really the only two scripts that needed porting).
2) Told the charm to install mariadb packages from universe on xenial. (This has the side effect of fixing an issue installing on z Linux -- if you'd like to avoid using the universe packages, I can change the check to install from universe only if we're on z Linux.)

To post a comment you must log in.
Pete Vander Giessen (petevg) wrote :

Just a quick note on this: part of what I did was sync up charm_helpers, to incorporate recent fixes that handle the case where Pyaml is not installed on the version of Python that you're using to execute a script (this makes the charm work on python3 in trusty).

That makes the PR look a lot bigger than it actually is.

Unmerged revisions

38. By Pete Vander Giessen <email address hidden> on 2016-07-12

Updated to be multi-series charm (trusty and xenial)

37. By Pete Vander Giessen <email address hidden> on 2016-07-08

Updated charm to work on xenial.

Python2 is not installed by default on xenial. Ported install and config-changed to python3 so that they will execute on a fresh xenial box. (The config-changed hook install python-mysql, which depends on python2, so the other hooks did not need to be ported).

Updated charmhelpers to latest version, so that missing dependencies, like Pyaml, will be installed (fixes an issue in trusty, where Pyaml is not installed by default in python3).

Charm now uses mariadb packages from universe in xenial, as those packages are up-to-date.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'README.md'
--- README.md 2016-06-03 17:18:00 +0000
+++ README.md 2016-07-12 15:38:59 +0000
@@ -11,78 +11,67 @@
11MariaDB and enhances it with an optimized configuration, additional testing,11MariaDB and enhances it with an optimized configuration, additional testing,
12and available 24/7 professional support and consulting.12and available 24/7 professional support and consulting.
1313
14This charm deploys either MariaDB, using packages provided by the MariaDB14This charm deploys MariaDB using packages in a repository provided by the
15Foundation, including packages for IBM's POWER8 platform; or MariaDB15MariaDB Foundation or, optionally, MariaDB Enterprise packages in a repository
16Enterprise, using packages in a repository provided by MariaDB Corporation,16provided by MariaDB Corporation, Inc.
17Inc.
1817
19Note: Packages for IBM's POWER8 platform are only available from the MariaDB18**Packages for IBM's Power8 platform are only available from the MariaDB
20Foundation repository.19Enterprise repository.**
2120
22As much as possible this charm uses the same charm structure as the MySQL charm21As much as possible this charm uses the same charm structure as the MySQL charm
23for the sake of compatability.22for the sake of compatability.
2423
24
25# Usage25# Usage
2626
27## General Usage27## General Usage
2828
29### Deploying MariaDB29To deploy a MariaDB service:
30To deploy MariaDB from the MariaDB Foundation, simply deploy like so:
3130
32 juju deploy mariadb31 juju deploy mariadb
3332
34MariaDB will be deployed.33Once deployed, you can retrive the MariaDB root user password by logging in to
3534the machine via `juju ssh` and reading the `/var/lib/mysql/mysql.passwd` file.
36### Deploying MariaDB Enterprise35To log in as root MariaDB User at the MariaDB console you can issue the
37To deploy a MariaDB Enterprise service, first login to the36following:
38[MariaDB Portal](https://mariadb.com/my_portal). On that page you will find37
39your MariaDB Enterprise download token. It is of the form `xxxx-xxxx`.38 juju ssh mariadb/0
39 mysql -u root -p$(sudo cat /var/lib/mysql/mysql.passwd)
40
41## To deploy MariaDB Enterprise instead of MariaDB
42
43First obtain a username/password from the [MariaDB Portal](http://mariadb.com)
44and you will then have access to the MariaDB Enterprise repository.
4045
41Next create a file called enterprise.yaml with the following contents,46Next create a file called enterprise.yaml with the following contents,
42replacing the placeholder token with your actual token:47replacing `username:password` with your actual username and password:
4348
44 mariadb:49 mariadb:
45 enterprise-eula: true50 enterprise-eula: true
46 token: "xxxx-xxxx"51 key: 0xd324876ebe6a595f
52 source: "deb https://username:password@code.mariadb.com/mariadb-enterprise/10.0/repo/ubuntu trusty main"
4753
48Lastly, deploy MariaDB Enterprise like so:54Lastly, deploy MariaDB as normal but with the addition of:
4955
50 juju deploy --config ./enterprise.yaml mariadb56 juju deploy --config ./enterprise.yaml mariadb
5157
52MariaDB Enterprise will be deployed. You must agree to all terms contained in58MariaDB Enterprise will be deployed instead of MariaDB. You must agree to all
59terms contained in `ENTERPRISE-LICENSE.md` in the charm directory to use
60MariaDB Enterprise.
61
62## To switch from MariaDB to MariaDB Enterprise
63
64If you deployed MariaDB and would like to switch to MariaDB Enterprise, first
65obtain a username/password from the [MariaDB Portal](http://mariadb.com) and
66you will then have access to the MariaDB Enterprise repository. You can then
67enable the repository in the charm with the following configuration:
68
69 juju set mariadb enterprise-eula=true key="0xd324876ebe6a595f" source="deb https://username:password@code.mariadb.com/mariadb-enterprise/10.0/repo/ubuntu trusty main"
70
71This will perform an in-place binary upgrade on all the MariaDB nodes from
72MariaDB to MariaDB Enterprise. You must agree to all terms contained in
53`ENTERPRISE-LICENSE.md` in the charm directory to use MariaDB Enterprise.73`ENTERPRISE-LICENSE.md` in the charm directory to use MariaDB Enterprise.
5474
55### Installing a different series of MariaDB
56Different series of MariaDB can be installed using the charm. The default
57series is MariaDB 10.1, but the older MariaDB 5.5 and 10.0 are also available.
58To install one of these older series, create a mariadb.yaml file with the
59following contents (example is for MariaDB 5.5):
60
61 mariadb:
62 series: "5.5"
63
64You could also add the series line to your enterprise.yaml file, if you are
65using MariaDB Enterprise. Then when deploying MariaDB, reference the file like
66so:
67
68 juju deploy --config ./mariadb.yaml mariadb
69
70Warning: Using the set command to downgrade MariaDB after initial deployment, for example, like so:
71
72 juju set mariadb series="5.5"
73
74...does not work. Using the set command to upgrade MariaDB; from 5.5 to 10.0,
75or from 10.0 to 10.1; does work.
76
77
78### After deploying
79Once deployed, you can retrive the MariaDB root user password by logging in to
80the machine via `juju ssh` and reading the `/var/lib/mysql/mysql.passwd` file.
81To log in as the root MariaDB user at the MariaDB console you can, for example,
82issue the following:
83
84 juju ssh mariadb/0
85 mysql -u root -p$(sudo cat /var/lib/mysql/mysql.passwd)
8675
87# Scale Out Usage76# Scale Out Usage
8877
@@ -96,7 +85,7 @@
96To deploy a slave:85To deploy a slave:
9786
98 # deploy second service87 # deploy second service
99 juju deploy --config ./enterprise.yaml mariadb mariadb-slave88 juju deploy mariadb mariadb-slave
10089
101 # add master to slave relation90 # add master to slave relation
102 juju add-relation mariadb:master mariadb-slave:slave91 juju add-relation mariadb:master mariadb-slave:slave
10392
=== modified file 'config.yaml'
--- config.yaml 2016-06-03 16:06:28 +0000
+++ config.yaml 2016-07-12 15:38:59 +0000
@@ -1,6 +1,6 @@
1options:1options:
2 dataset-size:2 dataset-size:
3 default: '50%'3 default: '80%'
4 description: |4 description: |
5 How much data do you want to keep in memory in the database. This5 How much data do you want to keep in memory in the database. This
6 will be used to tune settings in the database server appropriately.6 will be used to tune settings in the database server appropriately.
@@ -107,38 +107,13 @@
107 I have read and agree to the ENTERPRISE TRIAL agreement, located107 I have read and agree to the ENTERPRISE TRIAL agreement, located
108 in ENTERPRISE-LICENSE.md located in the charm, or on the web here:108 in ENTERPRISE-LICENSE.md located in the charm, or on the web here:
109 https://mariadb.com/about/legal/evaluation-agreement109 https://mariadb.com/about/legal/evaluation-agreement
110 base-url:110 source:
111 type: string111 type: string
112 default: "https://downloads.mariadb.com/enterprise"112 default: "deb http://ftp.osuosl.org/pub/mariadb/repo/10.0/ubuntu trusty main"
113 description: |113 description: |
114 Base URL of the MariaDB Enterprise repository package114 Repository Mirror string to install MariaDB from
115 base-url-org:
116 type: string
117 default: "http://ftp.osuosl.org/pub/mariadb/repo"
118 description: |
119 Base URL of the MariaDB repository
120 series:
121 type: string
122 default: "10.1"
123 description: |
124 Name of the MariaDB series to install
125 token:
126 type: string
127 default: ""
128 description: |
129 Enterprise download token from https://mariadb.com/my_portal
130 repo-pkg:
131 type: string
132 default: "mariadb-enterprise-repository.deb"
133 description: |
134 The name of the MariaDB Enterprise repository package
135 key:115 key:
136 type: string116 type: string
137 default: "0xce1a3dd5e3c94f49"117 default: "0xcbcb082a1bb943db"
138 description: |118 description: |
139 GPG Key used to verify MariaDB Enterprise packages119 GPG Key used to verify apt packages.
140 key-org:
141 type: string
142 default: "0xcbcb082a1bb943db 0xF1656F24C74CD1D8"
143 description: |
144 GPG Keys used to verify MariaDB packages
145120
=== modified file 'hooks/common.py'
--- hooks/common.py 2014-09-25 20:40:27 +0000
+++ hooks/common.py 2016-07-12 15:38:59 +0000
@@ -76,7 +76,7 @@
76def create_database(db_name):76def create_database(db_name):
77 cursor = get_db_cursor()77 cursor = get_db_cursor()
78 try:78 try:
79 cursor.execute("CREATE DATABASE {}".format(db_name))79 cursor.execute("CREATE DATABASE `{}`".format(db_name))
80 finally:80 finally:
81 cursor.close()81 cursor.close()
8282
@@ -99,7 +99,7 @@
99 remote_ip, password):99 remote_ip, password):
100 cursor = get_db_cursor()100 cursor = get_db_cursor()
101 try:101 try:
102 cursor.execute("GRANT ALL PRIVILEGES ON {}.* TO '{}'@'{}' "\102 cursor.execute("GRANT ALL PRIVILEGES ON `{}`.* TO '{}'@'{}' "\
103 "IDENTIFIED BY '{}'".format(db_name,103 "IDENTIFIED BY '{}'".format(db_name,
104 db_user,104 db_user,
105 remote_ip,105 remote_ip,
106106
=== modified file 'hooks/config-changed'
--- hooks/config-changed 2016-05-31 20:31:40 +0000
+++ hooks/config-changed 2016-07-12 15:38:59 +0000
@@ -1,4 +1,4 @@
1#!/usr/bin/python1#!/usr/bin/env python3
22
3from subprocess import check_output, check_call, CalledProcessError3from subprocess import check_output, check_call, CalledProcessError
4import tempfile4import tempfile
@@ -8,17 +8,12 @@
8import os8import os
9import sys9import sys
10import platform10import platform
11from string import upper
12from subprocess import Popen, PIPE11from subprocess import Popen, PIPE
1312
14sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))13sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))
1514
16from charmhelpers import fetch15from charmhelpers import fetch
1716
18from charmhelpers.core.host import (
19 lsb_release
20)
21
22from charmhelpers.core import (17from charmhelpers.core import (
23 hookenv,18 hookenv,
24 host,19 host,
@@ -67,7 +62,8 @@
67if IS_32BIT_SYSTEM:62if IS_32BIT_SYSTEM:
68 log("32bit system restrictions in play", "INFO")63 log("32bit system restrictions in play", "INFO")
6964
70configs = json.loads(check_output(['config-get','--format=json']))65configs = json.loads(
66 check_output(['config-get','--format=json']).decode('utf-8'))
7167
72def get_memtotal():68def get_memtotal():
73 with open('/proc/meminfo') as meminfo_file:69 with open('/proc/meminfo') as meminfo_file:
@@ -75,19 +71,13 @@
75 (key, mem) = line.split(':', 2)71 (key, mem) = line.split(':', 2)
76 if key == 'MemTotal':72 if key == 'MemTotal':
77 (mtot, modifier) = mem.strip().split(' ')73 (mtot, modifier) = mem.strip().split(' ')
78 return '%s%s' % (mtot, upper(modifier[0]))74 return '%s%s' % (mtot, modifier[0].upper())
7975
8076
8177
82#source = config['source']78source = config['source']
83accepted = config['enterprise-eula']79accepted = config['enterprise-eula']
84key = config['key']80_, _, series = platform.dist()
85key_org = config['key-org']
86base = config['base-url']
87base_org = config['base-url-org']
88token = config['token']
89pkg = config['repo-pkg']
90series = config['series']
9181
9282
93# set MariaDB Root Password83# set MariaDB Root Password
@@ -97,31 +87,20 @@
9787
98# preseed debconf with our admin password88# preseed debconf with our admin password
99dconf = Popen(['debconf-set-selections'], stdin=PIPE)89dconf = Popen(['debconf-set-selections'], stdin=PIPE)
100dconf.stdin.write("%s %s/root_password password %s\n" % (package, package, root_pass))90dconf.stdin.write(("%s %s/root_password password %s\n" % (package, package, root_pass)).encode())
101dconf.stdin.write("%s %s/root_password_again password %s\n" % (package, package, root_pass))91dconf.stdin.write(("%s %s/root_password_again password %s\n" % (package, package, root_pass)).encode())
102dconf.communicate()92dconf.communicate()
103dconf.wait()93dconf.wait()
10494
105# assumption of mariadb packages being delivered from code.mariadb95# assumption of mariadb packages being delivered from code.mariadb
106if not accepted:96if not accepted and "code.mariadb" in source:
107 log('Enterprise EULA not accepted - installing from mariadb.org', 'INFO')97 log('EULA not accepted - doing nothing', 'WARNING')
108 check_call(['apt-key', 'adv', '--keyserver',98 host.service_stop('mysql')
109 'hkp://keyserver.ubuntu.com:80',
110 '--recv', key_org])
111 release = lsb_release()['DISTRIB_CODENAME']
112 fetch.add_source("deb %s/%s/ubuntu %s main" % (base_org,series,release), None)
113 fetch.apt_update()
114 packages = ['mariadb-server', 'mariadb-client']
115 fetch.apt_install(packages)
116
117else:99else:
118 check_call(['apt-key', 'adv', '--keyserver',100 if series == 'trusty':
119 'hkp://keyserver.ubuntu.com:80',101 # an up-to-date mariadb is in universe on xenial and newer.
120 '--recv', key])102 fetch.add_source(source, config['key'])
121 check_call(['wget', '-N', 103 fetch.apt_update()
122 "%s/%s/generate/%s/%s" % (base, token, series, pkg)])
123 check_call(['dpkg', '-i', 'mariadb-enterprise-repository.deb'])
124 fetch.apt_update()
125104
126 packages = ['mariadb-server', 'mariadb-client']105 packages = ['mariadb-server', 'mariadb-client']
127 fetch.apt_install(packages)106 fetch.apt_install(packages)
@@ -234,7 +213,7 @@
234#213#
235# * Fine Tuning214# * Fine Tuning
236#215#
237key_buffer = %(key-buffer)s216key_buffer = %(key-buffer)s
238max_allowed_packet = 16M217max_allowed_packet = 16M
239thread_stack = 192K218thread_stack = 192K
240thread_cache_size = 8219thread_cache_size = 8
@@ -258,9 +237,9 @@
258# As of 5.1 you can enable the log at runtime!237# As of 5.1 you can enable the log at runtime!
259#general_log_file = /usr/local/mysql/data/mysql.log238#general_log_file = /usr/local/mysql/data/mysql.log
260#general_log = 1239#general_log = 1
261# 240#
262# Error log - should be very few entries.241# Error log - should be very few entries.
263# 242#
264log_error = /var/log/mysql/error.log243log_error = /var/log/mysql/error.log
265#244#
266# Here you can see queries with especially long duration245# Here you can see queries with especially long duration
@@ -310,7 +289,7 @@
310#no-auto-rehash # faster start of mysql but no tab completition289#no-auto-rehash # faster start of mysql but no tab completition
311290
312[isamchk]291[isamchk]
313key_buffer = 16M292key_buffer = 16M
314293
315#294#
316# * IMPORTANT: Additional settings that can override those from this file!295# * IMPORTANT: Additional settings that can override those from this file!
@@ -343,7 +322,7 @@
343 }322 }
344323
345need_restart = False324need_restart = False
346for target, content in targets.iteritems():325for target, content in targets.items():
347 tdir = os.path.dirname(target)326 tdir = os.path.dirname(target)
348 if len(content) == 0 and os.path.exists(target):327 if len(content) == 0 and os.path.exists(target):
349 os.unlink(target)328 os.unlink(target)
@@ -353,11 +332,11 @@
353 t.write(content)332 t.write(content)
354 t.flush()333 t.flush()
355 tmd5 = hashlib.md5()334 tmd5 = hashlib.md5()
356 tmd5.update(content)335 tmd5.update(content.encode())
357 if os.path.exists(target):336 if os.path.exists(target):
358 with open(target, 'r') as old:337 with open(target, 'r') as old:
359 md5 = hashlib.md5()338 md5 = hashlib.md5()
360 md5.update(old.read())339 md5.update(old.read().encode())
361 oldhash = md5.digest()340 oldhash = md5.digest()
362 if oldhash != tmd5.digest():341 if oldhash != tmd5.digest():
363 os.rename(target, '%s.%s' % (target, md5.hexdigest()))342 os.rename(target, '%s.%s' % (target, md5.hexdigest()))
364343
=== modified file 'hooks/install'
--- hooks/install 2015-01-30 23:46:18 +0000
+++ hooks/install 2016-07-12 15:38:59 +0000
@@ -1,4 +1,4 @@
1#!/usr/bin/env python1#!/usr/bin/env python3
22
3import os3import os
4import sys4import sys
@@ -37,7 +37,7 @@
37 os.makedirs(varpath)37 os.makedirs(varpath)
38 with open(PASSFILE, 'a'):38 with open(PASSFILE, 'a'):
39 os.utime(PASSFILE, None)39 os.utime(PASSFILE, None)
40 os.chmod(PASSFILE, 0600)40 os.chmod(PASSFILE, 0o600)
41 except:41 except:
42 pass42 pass
43 # Touch the passfile43 # Touch the passfile
4444
=== modified file 'hooks/start'
--- hooks/start 2014-09-24 21:03:47 +0000
+++ hooks/start 2016-07-12 15:38:59 +0000
@@ -1,4 +1,5 @@
1#!/bin/bash1#!/bin/bash
2set -e2set -e
3/etc/init.d/mysql restart || /etc/init.d/mysql start3service mysql restart || service mysql start
4
45
56
=== modified file 'lib/charmhelpers/__init__.py'
--- lib/charmhelpers/__init__.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/__init__.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,38 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
17# Bootstrap charm-helpers, installing its dependencies if necessary using
18# only standard libraries.
19import subprocess
20import sys
21
22try:
23 import six # flake8: noqa
24except ImportError:
25 if sys.version_info.major == 2:
26 subprocess.check_call(['apt-get', 'install', '-y', 'python-six'])
27 else:
28 subprocess.check_call(['apt-get', 'install', '-y', 'python3-six'])
29 import six # flake8: noqa
30
31try:
32 import yaml # flake8: noqa
33except ImportError:
34 if sys.version_info.major == 2:
35 subprocess.check_call(['apt-get', 'install', '-y', 'python-yaml'])
36 else:
37 subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
38 import yaml # flake8: noqa
039
=== modified file 'lib/charmhelpers/core/__init__.py'
--- lib/charmhelpers/core/__init__.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/__init__.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,15 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
016
=== added file 'lib/charmhelpers/core/decorators.py'
--- lib/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
+++ lib/charmhelpers/core/decorators.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,57 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
17#
18# Copyright 2014 Canonical Ltd.
19#
20# Authors:
21# Edward Hope-Morley <opentastic@gmail.com>
22#
23
24import time
25
26from charmhelpers.core.hookenv import (
27 log,
28 INFO,
29)
30
31
32def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
33 """If the decorated function raises exception exc_type, allow num_retries
34 retry attempts before raise the exception.
35 """
36 def _retry_on_exception_inner_1(f):
37 def _retry_on_exception_inner_2(*args, **kwargs):
38 retries = num_retries
39 multiplier = 1
40 while True:
41 try:
42 return f(*args, **kwargs)
43 except exc_type:
44 if not retries:
45 raise
46
47 delay = base_delay * multiplier
48 multiplier += 1
49 log("Retrying '%s' %d more times (delay=%s)" %
50 (f.__name__, retries, delay), level=INFO)
51 retries -= 1
52 if delay:
53 time.sleep(delay)
54
55 return _retry_on_exception_inner_2
56
57 return _retry_on_exception_inner_1
058
=== added file 'lib/charmhelpers/core/files.py'
--- lib/charmhelpers/core/files.py 1970-01-01 00:00:00 +0000
+++ lib/charmhelpers/core/files.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,45 @@
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4# Copyright 2014-2015 Canonical Limited.
5#
6# This file is part of charm-helpers.
7#
8# charm-helpers is free software: you can redistribute it and/or modify
9# it under the terms of the GNU Lesser General Public License version 3 as
10# published by the Free Software Foundation.
11#
12# charm-helpers is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU Lesser General Public License for more details.
16#
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
19
20__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'
21
22import os
23import subprocess
24
25
26def sed(filename, before, after, flags='g'):
27 """
28 Search and replaces the given pattern on filename.
29
30 :param filename: relative or absolute file path.
31 :param before: expression to be replaced (see 'man sed')
32 :param after: expression to replace with (see 'man sed')
33 :param flags: sed-compatible regex flags in example, to make
34 the search and replace case insensitive, specify ``flags="i"``.
35 The ``g`` flag is always specified regardless, so you do not
36 need to remember to include it when overriding this parameter.
37 :returns: If the sed command exit code was zero then return,
38 otherwise raise CalledProcessError.
39 """
40 expression = r's/{0}/{1}/{2}'.format(before,
41 after, flags)
42
43 return subprocess.check_call(["sed", "-i", "-r", "-e",
44 expression,
45 os.path.expanduser(filename)])
046
=== modified file 'lib/charmhelpers/core/fstab.py'
--- lib/charmhelpers/core/fstab.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/fstab.py 2016-07-12 15:38:59 +0000
@@ -1,11 +1,27 @@
1#!/usr/bin/env python1#!/usr/bin/env python
2# -*- coding: utf-8 -*-2# -*- coding: utf-8 -*-
33
4__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'4# Copyright 2014-2015 Canonical Limited.
5#
6# This file is part of charm-helpers.
7#
8# charm-helpers is free software: you can redistribute it and/or modify
9# it under the terms of the GNU Lesser General Public License version 3 as
10# published by the Free Software Foundation.
11#
12# charm-helpers is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU Lesser General Public License for more details.
16#
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
519
6import io20import io
7import os21import os
822
23__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
24
925
10class Fstab(io.FileIO):26class Fstab(io.FileIO):
11 """This class extends file in order to implement a file reader/writer27 """This class extends file in order to implement a file reader/writer
@@ -61,7 +77,7 @@
61 for line in self.readlines():77 for line in self.readlines():
62 line = line.decode('us-ascii')78 line = line.decode('us-ascii')
63 try:79 try:
64 if line.strip() and not line.startswith("#"):80 if line.strip() and not line.strip().startswith("#"):
65 yield self._hydrate_entry(line)81 yield self._hydrate_entry(line)
66 except ValueError:82 except ValueError:
67 pass83 pass
@@ -88,7 +104,7 @@
88104
89 found = False105 found = False
90 for index, line in enumerate(lines):106 for index, line in enumerate(lines):
91 if not line.startswith("#"):107 if line.strip() and not line.strip().startswith("#"):
92 if self._hydrate_entry(line) == entry:108 if self._hydrate_entry(line) == entry:
93 found = True109 found = True
94 break110 break
95111
=== modified file 'lib/charmhelpers/core/hookenv.py'
--- lib/charmhelpers/core/hookenv.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/hookenv.py 2016-07-12 15:38:59 +0000
@@ -1,14 +1,37 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1"Interactions with the Juju environment"17"Interactions with the Juju environment"
2# Copyright 2013 Canonical Ltd.18# Copyright 2013 Canonical Ltd.
3#19#
4# Authors:20# Authors:
5# Charm Helpers Developers <juju@lists.ubuntu.com>21# Charm Helpers Developers <juju@lists.ubuntu.com>
622
23from __future__ import print_function
24import copy
25from distutils.version import LooseVersion
26from functools import wraps
27import glob
7import os28import os
8import json29import json
9import yaml30import yaml
10import subprocess31import subprocess
11import sys32import sys
33import errno
34import tempfile
12from subprocess import CalledProcessError35from subprocess import CalledProcessError
1336
14import six37import six
@@ -40,15 +63,18 @@
4063
41 will cache the result of unit_get + 'test' for future calls.64 will cache the result of unit_get + 'test' for future calls.
42 """65 """
66 @wraps(func)
43 def wrapper(*args, **kwargs):67 def wrapper(*args, **kwargs):
44 global cache68 global cache
45 key = str((func, args, kwargs))69 key = str((func, args, kwargs))
46 try:70 try:
47 return cache[key]71 return cache[key]
48 except KeyError:72 except KeyError:
49 res = func(*args, **kwargs)73 pass # Drop out of the exception handler scope.
50 cache[key] = res74 res = func(*args, **kwargs)
51 return res75 cache[key] = res
76 return res
77 wrapper._wrapped = func
52 return wrapper78 return wrapper
5379
5480
@@ -68,8 +94,21 @@
68 command = ['juju-log']94 command = ['juju-log']
69 if level:95 if level:
70 command += ['-l', level]96 command += ['-l', level]
97 if not isinstance(message, six.string_types):
98 message = repr(message)
71 command += [message]99 command += [message]
72 subprocess.call(command)100 # Missing juju-log should not cause failures in unit tests
101 # Send log output to stderr
102 try:
103 subprocess.call(command)
104 except OSError as e:
105 if e.errno == errno.ENOENT:
106 if level:
107 message = "{}: {}".format(level, message)
108 message = "juju-log: {}".format(message)
109 print(message, file=sys.stderr)
110 else:
111 raise
73112
74113
75class Serializable(UserDict):114class Serializable(UserDict):
@@ -135,9 +174,19 @@
135 return os.environ.get('JUJU_RELATION', None)174 return os.environ.get('JUJU_RELATION', None)
136175
137176
138def relation_id():177@cached
139 """The relation ID for the current relation hook"""178def relation_id(relation_name=None, service_or_unit=None):
140 return os.environ.get('JUJU_RELATION_ID', None)179 """The relation ID for the current or a specified relation"""
180 if not relation_name and not service_or_unit:
181 return os.environ.get('JUJU_RELATION_ID', None)
182 elif relation_name and service_or_unit:
183 service_name = service_or_unit.split('/')[0]
184 for relid in relation_ids(relation_name):
185 remote_service = remote_service_name(relid)
186 if remote_service == service_name:
187 return relid
188 else:
189 raise ValueError('Must specify neither or both of relation_name and service_or_unit')
141190
142191
143def local_unit():192def local_unit():
@@ -147,7 +196,7 @@
147196
148def remote_unit():197def remote_unit():
149 """The remote unit for the current relation hook"""198 """The remote unit for the current relation hook"""
150 return os.environ['JUJU_REMOTE_UNIT']199 return os.environ.get('JUJU_REMOTE_UNIT', None)
151200
152201
153def service_name():202def service_name():
@@ -155,9 +204,20 @@
155 return local_unit().split('/')[0]204 return local_unit().split('/')[0]
156205
157206
207@cached
208def remote_service_name(relid=None):
209 """The remote service name for a given relation-id (or the current relation)"""
210 if relid is None:
211 unit = remote_unit()
212 else:
213 units = related_units(relid)
214 unit = units[0] if units else None
215 return unit.split('/')[0] if unit else None
216
217
158def hook_name():218def hook_name():
159 """The name of the currently executing hook"""219 """The name of the currently executing hook"""
160 return os.path.basename(sys.argv[0])220 return os.environ.get('JUJU_HOOK_NAME', os.path.basename(sys.argv[0]))
161221
162222
163class Config(dict):223class Config(dict):
@@ -207,23 +267,7 @@
207 self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)267 self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
208 if os.path.exists(self.path):268 if os.path.exists(self.path):
209 self.load_previous()269 self.load_previous()
210270 atexit(self._implicit_save)
211 def __getitem__(self, key):
212 """For regular dict lookups, check the current juju config first,
213 then the previous (saved) copy. This ensures that user-saved values
214 will be returned by a dict lookup.
215
216 """
217 try:
218 return dict.__getitem__(self, key)
219 except KeyError:
220 return (self._prev_dict or {})[key]
221
222 def keys(self):
223 prev_keys = []
224 if self._prev_dict is not None:
225 prev_keys = self._prev_dict.keys()
226 return list(set(prev_keys + list(dict.keys(self))))
227271
228 def load_previous(self, path=None):272 def load_previous(self, path=None):
229 """Load previous copy of config from disk.273 """Load previous copy of config from disk.
@@ -242,6 +286,9 @@
242 self.path = path or self.path286 self.path = path or self.path
243 with open(self.path) as f:287 with open(self.path) as f:
244 self._prev_dict = json.load(f)288 self._prev_dict = json.load(f)
289 for k, v in copy.deepcopy(self._prev_dict).items():
290 if k not in self:
291 self[k] = v
245292
246 def changed(self, key):293 def changed(self, key):
247 """Return True if the current value for this key is different from294 """Return True if the current value for this key is different from
@@ -273,13 +320,13 @@
273 instance.320 instance.
274321
275 """322 """
276 if self._prev_dict:
277 for k, v in six.iteritems(self._prev_dict):
278 if k not in self:
279 self[k] = v
280 with open(self.path, 'w') as f:323 with open(self.path, 'w') as f:
281 json.dump(self, f)324 json.dump(self, f)
282325
326 def _implicit_save(self):
327 if self.implicit_save:
328 self.save()
329
283330
284@cached331@cached
285def config(scope=None):332def config(scope=None):
@@ -322,18 +369,49 @@
322 """Set relation information for the current unit"""369 """Set relation information for the current unit"""
323 relation_settings = relation_settings if relation_settings else {}370 relation_settings = relation_settings if relation_settings else {}
324 relation_cmd_line = ['relation-set']371 relation_cmd_line = ['relation-set']
372 accepts_file = "--file" in subprocess.check_output(
373 relation_cmd_line + ["--help"], universal_newlines=True)
325 if relation_id is not None:374 if relation_id is not None:
326 relation_cmd_line.extend(('-r', relation_id))375 relation_cmd_line.extend(('-r', relation_id))
327 for k, v in (list(relation_settings.items()) + list(kwargs.items())):376 settings = relation_settings.copy()
328 if v is None:377 settings.update(kwargs)
329 relation_cmd_line.append('{}='.format(k))378 for key, value in settings.items():
330 else:379 # Force value to be a string: it always should, but some call
331 relation_cmd_line.append('{}={}'.format(k, v))380 # sites pass in things like dicts or numbers.
332 subprocess.check_call(relation_cmd_line)381 if value is not None:
382 settings[key] = "{}".format(value)
383 if accepts_file:
384 # --file was introduced in Juju 1.23.2. Use it by default if
385 # available, since otherwise we'll break if the relation data is
386 # too big. Ideally we should tell relation-set to read the data from
387 # stdin, but that feature is broken in 1.23.2: Bug #1454678.
388 with tempfile.NamedTemporaryFile(delete=False) as settings_file:
389 settings_file.write(yaml.safe_dump(settings).encode("utf-8"))
390 subprocess.check_call(
391 relation_cmd_line + ["--file", settings_file.name])
392 os.remove(settings_file.name)
393 else:
394 for key, value in settings.items():
395 if value is None:
396 relation_cmd_line.append('{}='.format(key))
397 else:
398 relation_cmd_line.append('{}={}'.format(key, value))
399 subprocess.check_call(relation_cmd_line)
333 # Flush cache of any relation-gets for local unit400 # Flush cache of any relation-gets for local unit
334 flush(local_unit())401 flush(local_unit())
335402
336403
404def relation_clear(r_id=None):
405 ''' Clears any relation data already set on relation r_id '''
406 settings = relation_get(rid=r_id,
407 unit=local_unit())
408 for setting in settings:
409 if setting not in ['public-address', 'private-address']:
410 settings[setting] = None
411 relation_set(relation_id=r_id,
412 **settings)
413
414
337@cached415@cached
338def relation_ids(reltype=None):416def relation_ids(reltype=None):
339 """A list of relation_ids"""417 """A list of relation_ids"""
@@ -394,21 +472,101 @@
394472
395473
396@cached474@cached
475def metadata():
476 """Get the current charm metadata.yaml contents as a python object"""
477 with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
478 return yaml.safe_load(md)
479
480
481@cached
397def relation_types():482def relation_types():
398 """Get a list of relation types supported by this charm"""483 """Get a list of relation types supported by this charm"""
399 charmdir = os.environ.get('CHARM_DIR', '')
400 mdf = open(os.path.join(charmdir, 'metadata.yaml'))
401 md = yaml.safe_load(mdf)
402 rel_types = []484 rel_types = []
485 md = metadata()
403 for key in ('provides', 'requires', 'peers'):486 for key in ('provides', 'requires', 'peers'):
404 section = md.get(key)487 section = md.get(key)
405 if section:488 if section:
406 rel_types.extend(section.keys())489 rel_types.extend(section.keys())
407 mdf.close()
408 return rel_types490 return rel_types
409491
410492
411@cached493@cached
494def peer_relation_id():
495 '''Get the peers relation id if a peers relation has been joined, else None.'''
496 md = metadata()
497 section = md.get('peers')
498 if section:
499 for key in section:
500 relids = relation_ids(key)
501 if relids:
502 return relids[0]
503 return None
504
505
506@cached
507def relation_to_interface(relation_name):
508 """
509 Given the name of a relation, return the interface that relation uses.
510
511 :returns: The interface name, or ``None``.
512 """
513 return relation_to_role_and_interface(relation_name)[1]
514
515
516@cached
517def relation_to_role_and_interface(relation_name):
518 """
519 Given the name of a relation, return the role and the name of the interface
520 that relation uses (where role is one of ``provides``, ``requires``, or ``peers``).
521
522 :returns: A tuple containing ``(role, interface)``, or ``(None, None)``.
523 """
524 _metadata = metadata()
525 for role in ('provides', 'requires', 'peers'):
526 interface = _metadata.get(role, {}).get(relation_name, {}).get('interface')
527 if interface:
528 return role, interface
529 return None, None
530
531
532@cached
533def role_and_interface_to_relations(role, interface_name):
534 """
535 Given a role and interface name, return a list of relation names for the
536 current charm that use that interface under that role (where role is one
537 of ``provides``, ``requires``, or ``peers``).
538
539 :returns: A list of relation names.
540 """
541 _metadata = metadata()
542 results = []
543 for relation_name, relation in _metadata.get(role, {}).items():
544 if relation['interface'] == interface_name:
545 results.append(relation_name)
546 return results
547
548
549@cached
550def interface_to_relations(interface_name):
551 """
552 Given an interface, return a list of relation names for the current
553 charm that use that interface.
554
555 :returns: A list of relation names.
556 """
557 results = []
558 for role in ('provides', 'requires', 'peers'):
559 results.extend(role_and_interface_to_relations(role, interface_name))
560 return results
561
562
563@cached
564def charm_name():
565 """Get the name of the current charm as is specified on metadata.yaml"""
566 return metadata().get('name')
567
568
569@cached
412def relations():570def relations():
413 """Get a nested dictionary of relation data for all related units"""571 """Get a nested dictionary of relation data for all related units"""
414 rels = {}572 rels = {}
@@ -468,11 +626,48 @@
468 return None626 return None
469627
470628
629def unit_public_ip():
630 """Get this unit's public IP address"""
631 return unit_get('public-address')
632
633
471def unit_private_ip():634def unit_private_ip():
472 """Get this unit's private IP address"""635 """Get this unit's private IP address"""
473 return unit_get('private-address')636 return unit_get('private-address')
474637
475638
639@cached
640def storage_get(attribute=None, storage_id=None):
641 """Get storage attributes"""
642 _args = ['storage-get', '--format=json']
643 if storage_id:
644 _args.extend(('-s', storage_id))
645 if attribute:
646 _args.append(attribute)
647 try:
648 return json.loads(subprocess.check_output(_args).decode('UTF-8'))
649 except ValueError:
650 return None
651
652
653@cached
654def storage_list(storage_name=None):
655 """List the storage IDs for the unit"""
656 _args = ['storage-list', '--format=json']
657 if storage_name:
658 _args.append(storage_name)
659 try:
660 return json.loads(subprocess.check_output(_args).decode('UTF-8'))
661 except ValueError:
662 return None
663 except OSError as e:
664 import errno
665 if e.errno == errno.ENOENT:
666 # storage-list does not exist
667 return []
668 raise
669
670
476class UnregisteredHookError(Exception):671class UnregisteredHookError(Exception):
477 """Raised when an undefined hook is called"""672 """Raised when an undefined hook is called"""
478 pass673 pass
@@ -500,10 +695,14 @@
500 hooks.execute(sys.argv)695 hooks.execute(sys.argv)
501 """696 """
502697
503 def __init__(self, config_save=True):698 def __init__(self, config_save=None):
504 super(Hooks, self).__init__()699 super(Hooks, self).__init__()
505 self._hooks = {}700 self._hooks = {}
506 self._config_save = config_save701
702 # For unknown reasons, we allow the Hooks constructor to override
703 # config().implicit_save.
704 if config_save is not None:
705 config().implicit_save = config_save
507706
508 def register(self, name, function):707 def register(self, name, function):
509 """Register a hook"""708 """Register a hook"""
@@ -511,13 +710,16 @@
511710
512 def execute(self, args):711 def execute(self, args):
513 """Execute a registered hook based on args[0]"""712 """Execute a registered hook based on args[0]"""
713 _run_atstart()
514 hook_name = os.path.basename(args[0])714 hook_name = os.path.basename(args[0])
515 if hook_name in self._hooks:715 if hook_name in self._hooks:
516 self._hooks[hook_name]()716 try:
517 if self._config_save:717 self._hooks[hook_name]()
518 cfg = config()718 except SystemExit as x:
519 if cfg.implicit_save:719 if x.code is None or x.code == 0:
520 cfg.save()720 _run_atexit()
721 raise
722 _run_atexit()
521 else:723 else:
522 raise UnregisteredHookError(hook_name)724 raise UnregisteredHookError(hook_name)
523725
@@ -538,3 +740,270 @@
538def charm_dir():740def charm_dir():
539 """Return the root directory of the current charm"""741 """Return the root directory of the current charm"""
540 return os.environ.get('CHARM_DIR')742 return os.environ.get('CHARM_DIR')
743
744
745@cached
746def action_get(key=None):
747 """Gets the value of an action parameter, or all key/value param pairs"""
748 cmd = ['action-get']
749 if key is not None:
750 cmd.append(key)
751 cmd.append('--format=json')
752 action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
753 return action_data
754
755
756def action_set(values):
757 """Sets the values to be returned after the action finishes"""
758 cmd = ['action-set']
759 for k, v in list(values.items()):
760 cmd.append('{}={}'.format(k, v))
761 subprocess.check_call(cmd)
762
763
764def action_fail(message):
765 """Sets the action status to failed and sets the error message.
766
767 The results set by action_set are preserved."""
768 subprocess.check_call(['action-fail', message])
769
770
771def action_name():
772 """Get the name of the currently executing action."""
773 return os.environ.get('JUJU_ACTION_NAME')
774
775
776def action_uuid():
777 """Get the UUID of the currently executing action."""
778 return os.environ.get('JUJU_ACTION_UUID')
779
780
781def action_tag():
782 """Get the tag for the currently executing action."""
783 return os.environ.get('JUJU_ACTION_TAG')
784
785
786def status_set(workload_state, message):
787 """Set the workload state with a message
788
789 Use status-set to set the workload state with a message which is visible
790 to the user via juju status. If the status-set command is not found then
791 assume this is juju < 1.23 and juju-log the message unstead.
792
793 workload_state -- valid juju workload state.
794 message -- status update message
795 """
796 valid_states = ['maintenance', 'blocked', 'waiting', 'active']
797 if workload_state not in valid_states:
798 raise ValueError(
799 '{!r} is not a valid workload state'.format(workload_state)
800 )
801 cmd = ['status-set', workload_state, message]
802 try:
803 ret = subprocess.call(cmd)
804 if ret == 0:
805 return
806 except OSError as e:
807 if e.errno != errno.ENOENT:
808 raise
809 log_message = 'status-set failed: {} {}'.format(workload_state,
810 message)
811 log(log_message, level='INFO')
812
813
814def status_get():
815 """Retrieve the previously set juju workload state and message
816
817 If the status-get command is not found then assume this is juju < 1.23 and
818 return 'unknown', ""
819
820 """
821 cmd = ['status-get', "--format=json", "--include-data"]
822 try:
823 raw_status = subprocess.check_output(cmd)
824 except OSError as e:
825 if e.errno == errno.ENOENT:
826 return ('unknown', "")
827 else:
828 raise
829 else:
830 status = json.loads(raw_status.decode("UTF-8"))
831 return (status["status"], status["message"])
832
833
834def translate_exc(from_exc, to_exc):
835 def inner_translate_exc1(f):
836 @wraps(f)
837 def inner_translate_exc2(*args, **kwargs):
838 try:
839 return f(*args, **kwargs)
840 except from_exc:
841 raise to_exc
842
843 return inner_translate_exc2
844
845 return inner_translate_exc1
846
847
848@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
849def is_leader():
850 """Does the current unit hold the juju leadership
851
852 Uses juju to determine whether the current unit is the leader of its peers
853 """
854 cmd = ['is-leader', '--format=json']
855 return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
856
857
858@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
859def leader_get(attribute=None):
860 """Juju leader get value(s)"""
861 cmd = ['leader-get', '--format=json'] + [attribute or '-']
862 return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
863
864
865@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
866def leader_set(settings=None, **kwargs):
867 """Juju leader set value(s)"""
868 # Don't log secrets.
869 # log("Juju leader-set '%s'" % (settings), level=DEBUG)
870 cmd = ['leader-set']
871 settings = settings or {}
872 settings.update(kwargs)
873 for k, v in settings.items():
874 if v is None:
875 cmd.append('{}='.format(k))
876 else:
877 cmd.append('{}={}'.format(k, v))
878 subprocess.check_call(cmd)
879
880
881@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
882def payload_register(ptype, klass, pid):
883 """ is used while a hook is running to let Juju know that a
884 payload has been started."""
885 cmd = ['payload-register']
886 for x in [ptype, klass, pid]:
887 cmd.append(x)
888 subprocess.check_call(cmd)
889
890
891@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
892def payload_unregister(klass, pid):
893 """ is used while a hook is running to let Juju know
894 that a payload has been manually stopped. The <class> and <id> provided
895 must match a payload that has been previously registered with juju using
896 payload-register."""
897 cmd = ['payload-unregister']
898 for x in [klass, pid]:
899 cmd.append(x)
900 subprocess.check_call(cmd)
901
902
903@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
904def payload_status_set(klass, pid, status):
905 """is used to update the current status of a registered payload.
906 The <class> and <id> provided must match a payload that has been previously
907 registered with juju using payload-register. The <status> must be one of the
908 follow: starting, started, stopping, stopped"""
909 cmd = ['payload-status-set']
910 for x in [klass, pid, status]:
911 cmd.append(x)
912 subprocess.check_call(cmd)
913
914
915@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
916def resource_get(name):
917 """used to fetch the resource path of the given name.
918
919 <name> must match a name of defined resource in metadata.yaml
920
921 returns either a path or False if resource not available
922 """
923 if not name:
924 return False
925
926 cmd = ['resource-get', name]
927 try:
928 return subprocess.check_output(cmd).decode('UTF-8')
929 except subprocess.CalledProcessError:
930 return False
931
932
933@cached
934def juju_version():
935 """Full version string (eg. '1.23.3.1-trusty-amd64')"""
936 # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
937 jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
938 return subprocess.check_output([jujud, 'version'],
939 universal_newlines=True).strip()
940
941
942@cached
943def has_juju_version(minimum_version):
944 """Return True if the Juju version is at least the provided version"""
945 return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
946
947
948_atexit = []
949_atstart = []
950
951
952def atstart(callback, *args, **kwargs):
953 '''Schedule a callback to run before the main hook.
954
955 Callbacks are run in the order they were added.
956
957 This is useful for modules and classes to perform initialization
958 and inject behavior. In particular:
959
960 - Run common code before all of your hooks, such as logging
961 the hook name or interesting relation data.
962 - Defer object or module initialization that requires a hook
963 context until we know there actually is a hook context,
964 making testing easier.
965 - Rather than requiring charm authors to include boilerplate to
966 invoke your helper's behavior, have it run automatically if
967 your object is instantiated or module imported.
968
969 This is not at all useful after your hook framework as been launched.
970 '''
971 global _atstart
972 _atstart.append((callback, args, kwargs))
973
974
975def atexit(callback, *args, **kwargs):
976 '''Schedule a callback to run on successful hook completion.
977
978 Callbacks are run in the reverse order that they were added.'''
979 _atexit.append((callback, args, kwargs))
980
981
982def _run_atstart():
983 '''Hook frameworks must invoke this before running the main hook body.'''
984 global _atstart
985 for callback, args, kwargs in _atstart:
986 callback(*args, **kwargs)
987 del _atstart[:]
988
989
990def _run_atexit():
991 '''Hook frameworks must invoke this after the main hook body has
992 successfully completed. Do not invoke it if the hook fails.'''
993 global _atexit
994 for callback, args, kwargs in reversed(_atexit):
995 callback(*args, **kwargs)
996 del _atexit[:]
997
998
999@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1000def network_get_primary_address(binding):
1001 '''
1002 Retrieve the primary network address for a named binding
1003
1004 :param binding: string. The name of a relation of extra-binding
1005 :return: string. The primary IP address for the named binding
1006 :raise: NotImplementedError if run on Juju < 2.0
1007 '''
1008 cmd = ['network-get', '--primary-address', binding]
1009 return subprocess.check_output(cmd).decode('UTF-8').strip()
5411010
=== modified file 'lib/charmhelpers/core/host.py'
--- lib/charmhelpers/core/host.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/host.py 2016-07-12 15:38:59 +0000
@@ -1,3 +1,19 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1"""Tools for working with the host system"""17"""Tools for working with the host system"""
2# Copyright 2012 Canonical Ltd.18# Copyright 2012 Canonical Ltd.
3#19#
@@ -8,11 +24,14 @@
8import os24import os
9import re25import re
10import pwd26import pwd
27import glob
11import grp28import grp
12import random29import random
13import string30import string
14import subprocess31import subprocess
15import hashlib32import hashlib
33import functools
34import itertools
16from contextlib import contextmanager35from contextlib import contextmanager
17from collections import OrderedDict36from collections import OrderedDict
1837
@@ -46,25 +65,94 @@
46 return service_result65 return service_result
4766
4867
68def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"):
69 """Pause a system service.
70
71 Stop it, and prevent it from starting again at boot."""
72 stopped = True
73 if service_running(service_name):
74 stopped = service_stop(service_name)
75 upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
76 sysv_file = os.path.join(initd_dir, service_name)
77 if init_is_systemd():
78 service('disable', service_name)
79 elif os.path.exists(upstart_file):
80 override_path = os.path.join(
81 init_dir, '{}.override'.format(service_name))
82 with open(override_path, 'w') as fh:
83 fh.write("manual\n")
84 elif os.path.exists(sysv_file):
85 subprocess.check_call(["update-rc.d", service_name, "disable"])
86 else:
87 raise ValueError(
88 "Unable to detect {0} as SystemD, Upstart {1} or"
89 " SysV {2}".format(
90 service_name, upstart_file, sysv_file))
91 return stopped
92
93
94def service_resume(service_name, init_dir="/etc/init",
95 initd_dir="/etc/init.d"):
96 """Resume a system service.
97
98 Reenable starting again at boot. Start the service"""
99 upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
100 sysv_file = os.path.join(initd_dir, service_name)
101 if init_is_systemd():
102 service('enable', service_name)
103 elif os.path.exists(upstart_file):
104 override_path = os.path.join(
105 init_dir, '{}.override'.format(service_name))
106 if os.path.exists(override_path):
107 os.unlink(override_path)
108 elif os.path.exists(sysv_file):
109 subprocess.check_call(["update-rc.d", service_name, "enable"])
110 else:
111 raise ValueError(
112 "Unable to detect {0} as SystemD, Upstart {1} or"
113 " SysV {2}".format(
114 service_name, upstart_file, sysv_file))
115
116 started = service_running(service_name)
117 if not started:
118 started = service_start(service_name)
119 return started
120
121
49def service(action, service_name):122def service(action, service_name):
50 """Control a system service"""123 """Control a system service"""
51 cmd = ['service', service_name, action]124 if init_is_systemd():
125 cmd = ['systemctl', action, service_name]
126 else:
127 cmd = ['service', service_name, action]
52 return subprocess.call(cmd) == 0128 return subprocess.call(cmd) == 0
53129
54130
55def service_running(service):131_UPSTART_CONF = "/etc/init/{}.conf"
132_INIT_D_CONF = "/etc/init.d/{}"
133
134
135def service_running(service_name):
56 """Determine whether a system service is running"""136 """Determine whether a system service is running"""
57 try:137 if init_is_systemd():
58 output = subprocess.check_output(138 return service('is-active', service_name)
59 ['service', service, 'status'],139 else:
60 stderr=subprocess.STDOUT).decode('UTF-8')140 if os.path.exists(_UPSTART_CONF.format(service_name)):
61 except subprocess.CalledProcessError:141 try:
142 output = subprocess.check_output(
143 ['status', service_name],
144 stderr=subprocess.STDOUT).decode('UTF-8')
145 except subprocess.CalledProcessError:
146 return False
147 else:
148 # This works for upstart scripts where the 'service' command
149 # returns a consistent string to represent running 'start/running'
150 if "start/running" in output:
151 return True
152 elif os.path.exists(_INIT_D_CONF.format(service_name)):
153 # Check System V scripts init script return codes
154 return service('status', service_name)
62 return False155 return False
63 else:
64 if ("start/running" in output or "is running" in output):
65 return True
66 else:
67 return False
68156
69157
70def service_available(service_name):158def service_available(service_name):
@@ -74,19 +162,46 @@
74 ['service', service_name, 'status'],162 ['service', service_name, 'status'],
75 stderr=subprocess.STDOUT).decode('UTF-8')163 stderr=subprocess.STDOUT).decode('UTF-8')
76 except subprocess.CalledProcessError as e:164 except subprocess.CalledProcessError as e:
77 return 'unrecognized service' not in e.output165 return b'unrecognized service' not in e.output
78 else:166 else:
79 return True167 return True
80168
81169
82def adduser(username, password=None, shell='/bin/bash', system_user=False):170SYSTEMD_SYSTEM = '/run/systemd/system'
83 """Add a user to the system"""171
172
173def init_is_systemd():
174 """Return True if the host system uses systemd, False otherwise."""
175 return os.path.isdir(SYSTEMD_SYSTEM)
176
177
178def adduser(username, password=None, shell='/bin/bash', system_user=False,
179 primary_group=None, secondary_groups=None, uid=None):
180 """Add a user to the system.
181
182 Will log but otherwise succeed if the user already exists.
183
184 :param str username: Username to create
185 :param str password: Password for user; if ``None``, create a system user
186 :param str shell: The default shell for the user
187 :param bool system_user: Whether to create a login or system user
188 :param str primary_group: Primary group for user; defaults to username
189 :param list secondary_groups: Optional list of additional groups
190 :param int uid: UID for user being created
191
192 :returns: The password database entry struct, as returned by `pwd.getpwnam`
193 """
84 try:194 try:
85 user_info = pwd.getpwnam(username)195 user_info = pwd.getpwnam(username)
86 log('user {0} already exists!'.format(username))196 log('user {0} already exists!'.format(username))
197 if uid:
198 user_info = pwd.getpwuid(int(uid))
199 log('user with uid {0} already exists!'.format(uid))
87 except KeyError:200 except KeyError:
88 log('creating user {0}'.format(username))201 log('creating user {0}'.format(username))
89 cmd = ['useradd']202 cmd = ['useradd']
203 if uid:
204 cmd.extend(['--uid', str(uid)])
90 if system_user or password is None:205 if system_user or password is None:
91 cmd.append('--system')206 cmd.append('--system')
92 else:207 else:
@@ -95,19 +210,99 @@
95 '--shell', shell,210 '--shell', shell,
96 '--password', password,211 '--password', password,
97 ])212 ])
213 if not primary_group:
214 try:
215 grp.getgrnam(username)
216 primary_group = username # avoid "group exists" error
217 except KeyError:
218 pass
219 if primary_group:
220 cmd.extend(['-g', primary_group])
221 if secondary_groups:
222 cmd.extend(['-G', ','.join(secondary_groups)])
98 cmd.append(username)223 cmd.append(username)
99 subprocess.check_call(cmd)224 subprocess.check_call(cmd)
100 user_info = pwd.getpwnam(username)225 user_info = pwd.getpwnam(username)
101 return user_info226 return user_info
102227
103228
229def user_exists(username):
230 """Check if a user exists"""
231 try:
232 pwd.getpwnam(username)
233 user_exists = True
234 except KeyError:
235 user_exists = False
236 return user_exists
237
238
239def uid_exists(uid):
240 """Check if a uid exists"""
241 try:
242 pwd.getpwuid(uid)
243 uid_exists = True
244 except KeyError:
245 uid_exists = False
246 return uid_exists
247
248
249def group_exists(groupname):
250 """Check if a group exists"""
251 try:
252 grp.getgrnam(groupname)
253 group_exists = True
254 except KeyError:
255 group_exists = False
256 return group_exists
257
258
259def gid_exists(gid):
260 """Check if a gid exists"""
261 try:
262 grp.getgrgid(gid)
263 gid_exists = True
264 except KeyError:
265 gid_exists = False
266 return gid_exists
267
268
269def add_group(group_name, system_group=False, gid=None):
270 """Add a group to the system
271
272 Will log but otherwise succeed if the group already exists.
273
274 :param str group_name: group to create
275 :param bool system_group: Create system group
276 :param int gid: GID for user being created
277
278 :returns: The password database entry struct, as returned by `grp.getgrnam`
279 """
280 try:
281 group_info = grp.getgrnam(group_name)
282 log('group {0} already exists!'.format(group_name))
283 if gid:
284 group_info = grp.getgrgid(gid)
285 log('group with gid {0} already exists!'.format(gid))
286 except KeyError:
287 log('creating group {0}'.format(group_name))
288 cmd = ['addgroup']
289 if gid:
290 cmd.extend(['--gid', str(gid)])
291 if system_group:
292 cmd.append('--system')
293 else:
294 cmd.extend([
295 '--group',
296 ])
297 cmd.append(group_name)
298 subprocess.check_call(cmd)
299 group_info = grp.getgrnam(group_name)
300 return group_info
301
302
104def add_user_to_group(username, group):303def add_user_to_group(username, group):
105 """Add a user to a group"""304 """Add a user to a group"""
106 cmd = [305 cmd = ['gpasswd', '-a', username, group]
107 'gpasswd', '-a',
108 username,
109 group
110 ]
111 log("Adding user {} to group {}".format(username, group))306 log("Adding user {} to group {}".format(username, group))
112 subprocess.check_call(cmd)307 subprocess.check_call(cmd)
113308
@@ -142,35 +337,36 @@
142 uid = pwd.getpwnam(owner).pw_uid337 uid = pwd.getpwnam(owner).pw_uid
143 gid = grp.getgrnam(group).gr_gid338 gid = grp.getgrnam(group).gr_gid
144 realpath = os.path.abspath(path)339 realpath = os.path.abspath(path)
145 if os.path.exists(realpath):340 path_exists = os.path.exists(realpath)
146 if force and not os.path.isdir(realpath):341 if path_exists and force:
342 if not os.path.isdir(realpath):
147 log("Removing non-directory file {} prior to mkdir()".format(path))343 log("Removing non-directory file {} prior to mkdir()".format(path))
148 os.unlink(realpath)344 os.unlink(realpath)
149 else:345 os.makedirs(realpath, perms)
346 elif not path_exists:
150 os.makedirs(realpath, perms)347 os.makedirs(realpath, perms)
151 os.chown(realpath, uid, gid)348 os.chown(realpath, uid, gid)
349 os.chmod(realpath, perms)
152350
153351
154def write_file(path, content, owner='root', group='root', perms=0o444):352def write_file(path, content, owner='root', group='root', perms=0o444):
155 """Create or overwrite a file with the contents of a string"""353 """Create or overwrite a file with the contents of a byte string."""
156 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))354 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
157 uid = pwd.getpwnam(owner).pw_uid355 uid = pwd.getpwnam(owner).pw_uid
158 gid = grp.getgrnam(group).gr_gid356 gid = grp.getgrnam(group).gr_gid
159 with open(path, 'w') as target:357 with open(path, 'wb') as target:
160 os.fchown(target.fileno(), uid, gid)358 os.fchown(target.fileno(), uid, gid)
161 os.fchmod(target.fileno(), perms)359 os.fchmod(target.fileno(), perms)
162 target.write(content)360 target.write(content)
163361
164362
165def fstab_remove(mp):363def fstab_remove(mp):
166 """Remove the given mountpoint entry from /etc/fstab364 """Remove the given mountpoint entry from /etc/fstab"""
167 """
168 return Fstab.remove_by_mountpoint(mp)365 return Fstab.remove_by_mountpoint(mp)
169366
170367
171def fstab_add(dev, mp, fs, options=None):368def fstab_add(dev, mp, fs, options=None):
172 """Adds the given device entry to the /etc/fstab file369 """Adds the given device entry to the /etc/fstab file"""
173 """
174 return Fstab.add(dev, mp, fs, options=options)370 return Fstab.add(dev, mp, fs, options=options)
175371
176372
@@ -214,9 +410,19 @@
214 return system_mounts410 return system_mounts
215411
216412
413def fstab_mount(mountpoint):
414 """Mount filesystem using fstab"""
415 cmd_args = ['mount', mountpoint]
416 try:
417 subprocess.check_output(cmd_args)
418 except subprocess.CalledProcessError as e:
419 log('Error unmounting {}\n{}'.format(mountpoint, e.output))
420 return False
421 return True
422
423
217def file_hash(path, hash_type='md5'):424def file_hash(path, hash_type='md5'):
218 """425 """Generate a hash checksum of the contents of 'path' or None if not found.
219 Generate a hash checksum of the contents of 'path' or None if not found.
220426
221 :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,427 :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
222 such as md5, sha1, sha256, sha512, etc.428 such as md5, sha1, sha256, sha512, etc.
@@ -230,9 +436,22 @@
230 return None436 return None
231437
232438
439def path_hash(path):
440 """Generate a hash checksum of all files matching 'path'. Standard
441 wildcards like '*' and '?' are supported, see documentation for the 'glob'
442 module for more information.
443
444 :return: dict: A { filename: hash } dictionary for all matched files.
445 Empty if none found.
446 """
447 return {
448 filename: file_hash(filename)
449 for filename in glob.iglob(path)
450 }
451
452
233def check_hash(path, checksum, hash_type='md5'):453def check_hash(path, checksum, hash_type='md5'):
234 """454 """Validate a file using a cryptographic checksum.
235 Validate a file using a cryptographic checksum.
236455
237 :param str checksum: Value of the checksum used to validate the file.456 :param str checksum: Value of the checksum used to validate the file.
238 :param str hash_type: Hash algorithm used to generate `checksum`.457 :param str hash_type: Hash algorithm used to generate `checksum`.
@@ -247,46 +466,80 @@
247466
248467
249class ChecksumError(ValueError):468class ChecksumError(ValueError):
469 """A class derived from Value error to indicate the checksum failed."""
250 pass470 pass
251471
252472
253def restart_on_change(restart_map, stopstart=False):473def restart_on_change(restart_map, stopstart=False, restart_functions=None):
254 """Restart services based on configuration files changing474 """Restart services based on configuration files changing
255475
256 This function is used a decorator, for example::476 This function is used a decorator, for example::
257477
258 @restart_on_change({478 @restart_on_change({
259 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]479 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
480 '/etc/apache/sites-enabled/*': [ 'apache2' ]
260 })481 })
261 def ceph_client_changed():482 def config_changed():
262 pass # your code here483 pass # your code here
263484
264 In this example, the cinder-api and cinder-volume services485 In this example, the cinder-api and cinder-volume services
265 would be restarted if /etc/ceph/ceph.conf is changed by the486 would be restarted if /etc/ceph/ceph.conf is changed by the
266 ceph_client_changed function.487 ceph_client_changed function. The apache2 service would be
488 restarted if any file matching the pattern got changed, created
489 or removed. Standard wildcards are supported, see documentation
490 for the 'glob' module for more information.
491
492 @param restart_map: {path_file_name: [service_name, ...]
493 @param stopstart: DEFAULT false; whether to stop, start OR restart
494 @param restart_functions: nonstandard functions to use to restart services
495 {svc: func, ...}
496 @returns result from decorated function
267 """497 """
268 def wrap(f):498 def wrap(f):
269 def wrapped_f(*args):499 @functools.wraps(f)
270 checksums = {}500 def wrapped_f(*args, **kwargs):
271 for path in restart_map:501 return restart_on_change_helper(
272 checksums[path] = file_hash(path)502 (lambda: f(*args, **kwargs)), restart_map, stopstart,
273 f(*args)503 restart_functions)
274 restarts = []
275 for path in restart_map:
276 if checksums[path] != file_hash(path):
277 restarts += restart_map[path]
278 services_list = list(OrderedDict.fromkeys(restarts))
279 if not stopstart:
280 for service_name in services_list:
281 service('restart', service_name)
282 else:
283 for action in ['stop', 'start']:
284 for service_name in services_list:
285 service(action, service_name)
286 return wrapped_f504 return wrapped_f
287 return wrap505 return wrap
288506
289507
508def restart_on_change_helper(lambda_f, restart_map, stopstart=False,
509 restart_functions=None):
510 """Helper function to perform the restart_on_change function.
511
512 This is provided for decorators to restart services if files described
513 in the restart_map have changed after an invocation of lambda_f().
514
515 @param lambda_f: function to call.
516 @param restart_map: {file: [service, ...]}
517 @param stopstart: whether to stop, start or restart a service
518 @param restart_functions: nonstandard functions to use to restart services
519 {svc: func, ...}
520 @returns result of lambda_f()
521 """
522 if restart_functions is None:
523 restart_functions = {}
524 checksums = {path: path_hash(path) for path in restart_map}
525 r = lambda_f()
526 # create a list of lists of the services to restart
527 restarts = [restart_map[path]
528 for path in restart_map
529 if path_hash(path) != checksums[path]]
530 # create a flat list of ordered services without duplicates from lists
531 services_list = list(OrderedDict.fromkeys(itertools.chain(*restarts)))
532 if services_list:
533 actions = ('stop', 'start') if stopstart else ('restart',)
534 for service_name in services_list:
535 if service_name in restart_functions:
536 restart_functions[service_name](service_name)
537 else:
538 for action in actions:
539 service(action, service_name)
540 return r
541
542
290def lsb_release():543def lsb_release():
291 """Return /etc/lsb-release in a dict"""544 """Return /etc/lsb-release in a dict"""
292 d = {}545 d = {}
@@ -300,45 +553,105 @@
300def pwgen(length=None):553def pwgen(length=None):
301 """Generate a random pasword."""554 """Generate a random pasword."""
302 if length is None:555 if length is None:
556 # A random length is ok to use a weak PRNG
303 length = random.choice(range(35, 45))557 length = random.choice(range(35, 45))
304 alphanumeric_chars = [558 alphanumeric_chars = [
305 l for l in (string.ascii_letters + string.digits)559 l for l in (string.ascii_letters + string.digits)
306 if l not in 'l0QD1vAEIOUaeiou']560 if l not in 'l0QD1vAEIOUaeiou']
561 # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
562 # actual password
563 random_generator = random.SystemRandom()
307 random_chars = [564 random_chars = [
308 random.choice(alphanumeric_chars) for _ in range(length)]565 random_generator.choice(alphanumeric_chars) for _ in range(length)]
309 return(''.join(random_chars))566 return(''.join(random_chars))
310567
311568
312def list_nics(nic_type):569def is_phy_iface(interface):
313 '''Return a list of nics of given type(s)'''570 """Returns True if interface is not virtual, otherwise False."""
571 if interface:
572 sys_net = '/sys/class/net'
573 if os.path.isdir(sys_net):
574 for iface in glob.glob(os.path.join(sys_net, '*')):
575 if '/virtual/' in os.path.realpath(iface):
576 continue
577
578 if interface == os.path.basename(iface):
579 return True
580
581 return False
582
583
584def get_bond_master(interface):
585 """Returns bond master if interface is bond slave otherwise None.
586
587 NOTE: the provided interface is expected to be physical
588 """
589 if interface:
590 iface_path = '/sys/class/net/%s' % (interface)
591 if os.path.exists(iface_path):
592 if '/virtual/' in os.path.realpath(iface_path):
593 return None
594
595 master = os.path.join(iface_path, 'master')
596 if os.path.exists(master):
597 master = os.path.realpath(master)
598 # make sure it is a bond master
599 if os.path.exists(os.path.join(master, 'bonding')):
600 return os.path.basename(master)
601
602 return None
603
604
605def list_nics(nic_type=None):
606 """Return a list of nics of given type(s)"""
314 if isinstance(nic_type, six.string_types):607 if isinstance(nic_type, six.string_types):
315 int_types = [nic_type]608 int_types = [nic_type]
316 else:609 else:
317 int_types = nic_type610 int_types = nic_type
611
318 interfaces = []612 interfaces = []
319 for int_type in int_types:613 if nic_type:
320 cmd = ['ip', 'addr', 'show', 'label', int_type + '*']614 for int_type in int_types:
615 cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
616 ip_output = subprocess.check_output(cmd).decode('UTF-8')
617 ip_output = ip_output.split('\n')
618 ip_output = (line for line in ip_output if line)
619 for line in ip_output:
620 if line.split()[1].startswith(int_type):
621 matched = re.search('.*: (' + int_type +
622 r'[0-9]+\.[0-9]+)@.*', line)
623 if matched:
624 iface = matched.groups()[0]
625 else:
626 iface = line.split()[1].replace(":", "")
627
628 if iface not in interfaces:
629 interfaces.append(iface)
630 else:
631 cmd = ['ip', 'a']
321 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')632 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
322 ip_output = (line for line in ip_output if line)633 ip_output = (line.strip() for line in ip_output if line)
634
635 key = re.compile('^[0-9]+:\s+(.+):')
323 for line in ip_output:636 for line in ip_output:
324 if line.split()[1].startswith(int_type):637 matched = re.search(key, line)
325 matched = re.search('.*: (bond[0-9]+\.[0-9]+)@.*', line)638 if matched:
326 if matched:639 iface = matched.group(1)
327 interface = matched.groups()[0]640 iface = iface.partition("@")[0]
328 else:641 if iface not in interfaces:
329 interface = line.split()[1].replace(":", "")642 interfaces.append(iface)
330 interfaces.append(interface)
331643
332 return interfaces644 return interfaces
333645
334646
335def set_nic_mtu(nic, mtu):647def set_nic_mtu(nic, mtu):
336 '''Set MTU on a network interface'''648 """Set the Maximum Transmission Unit (MTU) on a network interface."""
337 cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]649 cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]
338 subprocess.check_call(cmd)650 subprocess.check_call(cmd)
339651
340652
341def get_nic_mtu(nic):653def get_nic_mtu(nic):
654 """Return the Maximum Transmission Unit (MTU) for a network interface."""
342 cmd = ['ip', 'addr', 'show', nic]655 cmd = ['ip', 'addr', 'show', nic]
343 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')656 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
344 mtu = ""657 mtu = ""
@@ -350,6 +663,7 @@
350663
351664
352def get_nic_hwaddr(nic):665def get_nic_hwaddr(nic):
666 """Return the Media Access Control (MAC) for a network interface."""
353 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]667 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
354 ip_output = subprocess.check_output(cmd).decode('UTF-8')668 ip_output = subprocess.check_output(cmd).decode('UTF-8')
355 hwaddr = ""669 hwaddr = ""
@@ -360,37 +674,91 @@
360674
361675
362def cmp_pkgrevno(package, revno, pkgcache=None):676def cmp_pkgrevno(package, revno, pkgcache=None):
363 '''Compare supplied revno with the revno of the installed package677 """Compare supplied revno with the revno of the installed package
364678
365 * 1 => Installed revno is greater than supplied arg679 * 1 => Installed revno is greater than supplied arg
366 * 0 => Installed revno is the same as supplied arg680 * 0 => Installed revno is the same as supplied arg
367 * -1 => Installed revno is less than supplied arg681 * -1 => Installed revno is less than supplied arg
368682
369 '''683 This function imports apt_cache function from charmhelpers.fetch if
684 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
685 you call this function, or pass an apt_pkg.Cache() instance.
686 """
370 import apt_pkg687 import apt_pkg
371 from charmhelpers.fetch import apt_cache
372 if not pkgcache:688 if not pkgcache:
689 from charmhelpers.fetch import apt_cache
373 pkgcache = apt_cache()690 pkgcache = apt_cache()
374 pkg = pkgcache[package]691 pkg = pkgcache[package]
375 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)692 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
376693
377694
378@contextmanager695@contextmanager
379def chdir(d):696def chdir(directory):
697 """Change the current working directory to a different directory for a code
698 block and return the previous directory after the block exits. Useful to
699 run commands from a specificed directory.
700
701 :param str directory: The directory path to change to for this context.
702 """
380 cur = os.getcwd()703 cur = os.getcwd()
381 try:704 try:
382 yield os.chdir(d)705 yield os.chdir(directory)
383 finally:706 finally:
384 os.chdir(cur)707 os.chdir(cur)
385708
386709
387def chownr(path, owner, group):710def chownr(path, owner, group, follow_links=True, chowntopdir=False):
711 """Recursively change user and group ownership of files and directories
712 in given path. Doesn't chown path itself by default, only its children.
713
714 :param str path: The string path to start changing ownership.
715 :param str owner: The owner string to use when looking up the uid.
716 :param str group: The group string to use when looking up the gid.
717 :param bool follow_links: Also Chown links if True
718 :param bool chowntopdir: Also chown path itself if True
719 """
388 uid = pwd.getpwnam(owner).pw_uid720 uid = pwd.getpwnam(owner).pw_uid
389 gid = grp.getgrnam(group).gr_gid721 gid = grp.getgrnam(group).gr_gid
722 if follow_links:
723 chown = os.chown
724 else:
725 chown = os.lchown
390726
727 if chowntopdir:
728 broken_symlink = os.path.lexists(path) and not os.path.exists(path)
729 if not broken_symlink:
730 chown(path, uid, gid)
391 for root, dirs, files in os.walk(path):731 for root, dirs, files in os.walk(path):
392 for name in dirs + files:732 for name in dirs + files:
393 full = os.path.join(root, name)733 full = os.path.join(root, name)
394 broken_symlink = os.path.lexists(full) and not os.path.exists(full)734 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
395 if not broken_symlink:735 if not broken_symlink:
396 os.chown(full, uid, gid)736 chown(full, uid, gid)
737
738
739def lchownr(path, owner, group):
740 """Recursively change user and group ownership of files and directories
741 in a given path, not following symbolic links. See the documentation for
742 'os.lchown' for more information.
743
744 :param str path: The string path to start changing ownership.
745 :param str owner: The owner string to use when looking up the uid.
746 :param str group: The group string to use when looking up the gid.
747 """
748 chownr(path, owner, group, follow_links=False)
749
750
751def get_total_ram():
752 """The total amount of system RAM in bytes.
753
754 This is what is reported by the OS, and may be overcommitted when
755 there are multiple containers hosted on the same machine.
756 """
757 with open('/proc/meminfo', 'r') as f:
758 for line in f.readlines():
759 if line:
760 key, value, unit = line.split()
761 if key == 'MemTotal:':
762 assert unit == 'kB', 'Unknown unit'
763 return int(value) * 1024 # Classic, not KiB.
764 raise NotImplementedError()
397765
=== added file 'lib/charmhelpers/core/hugepage.py'
--- lib/charmhelpers/core/hugepage.py 1970-01-01 00:00:00 +0000
+++ lib/charmhelpers/core/hugepage.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,71 @@
1# -*- coding: utf-8 -*-
2
3# Copyright 2014-2015 Canonical Limited.
4#
5# This file is part of charm-helpers.
6#
7# charm-helpers is free software: you can redistribute it and/or modify
8# it under the terms of the GNU Lesser General Public License version 3 as
9# published by the Free Software Foundation.
10#
11# charm-helpers is distributed in the hope that it will be useful,
12# but WITHOUT ANY WARRANTY; without even the implied warranty of
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14# GNU Lesser General Public License for more details.
15#
16# You should have received a copy of the GNU Lesser General Public License
17# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
18
19import yaml
20from charmhelpers.core import fstab
21from charmhelpers.core import sysctl
22from charmhelpers.core.host import (
23 add_group,
24 add_user_to_group,
25 fstab_mount,
26 mkdir,
27)
28from charmhelpers.core.strutils import bytes_from_string
29from subprocess import check_output
30
31
32def hugepage_support(user, group='hugetlb', nr_hugepages=256,
33 max_map_count=65536, mnt_point='/run/hugepages/kvm',
34 pagesize='2MB', mount=True, set_shmmax=False):
35 """Enable hugepages on system.
36
37 Args:
38 user (str) -- Username to allow access to hugepages to
39 group (str) -- Group name to own hugepages
40 nr_hugepages (int) -- Number of pages to reserve
41 max_map_count (int) -- Number of Virtual Memory Areas a process can own
42 mnt_point (str) -- Directory to mount hugepages on
43 pagesize (str) -- Size of hugepages
44 mount (bool) -- Whether to Mount hugepages
45 """
46 group_info = add_group(group)
47 gid = group_info.gr_gid
48 add_user_to_group(user, group)
49 if max_map_count < 2 * nr_hugepages:
50 max_map_count = 2 * nr_hugepages
51 sysctl_settings = {
52 'vm.nr_hugepages': nr_hugepages,
53 'vm.max_map_count': max_map_count,
54 'vm.hugetlb_shm_group': gid,
55 }
56 if set_shmmax:
57 shmmax_current = int(check_output(['sysctl', '-n', 'kernel.shmmax']))
58 shmmax_minsize = bytes_from_string(pagesize) * nr_hugepages
59 if shmmax_minsize > shmmax_current:
60 sysctl_settings['kernel.shmmax'] = shmmax_minsize
61 sysctl.create(yaml.dump(sysctl_settings), '/etc/sysctl.d/10-hugepage.conf')
62 mkdir(mnt_point, owner='root', group='root', perms=0o755, force=False)
63 lfstab = fstab.Fstab()
64 fstab_entry = lfstab.get_entry_by_attr('mountpoint', mnt_point)
65 if fstab_entry:
66 lfstab.remove_entry(fstab_entry)
67 entry = lfstab.Entry('nodev', mnt_point, 'hugetlbfs',
68 'mode=1770,gid={},pagesize={}'.format(gid, pagesize), 0, 0)
69 lfstab.add_entry(entry)
70 if mount:
71 fstab_mount(mnt_point)
072
=== added file 'lib/charmhelpers/core/kernel.py'
--- lib/charmhelpers/core/kernel.py 1970-01-01 00:00:00 +0000
+++ lib/charmhelpers/core/kernel.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,68 @@
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4# Copyright 2014-2015 Canonical Limited.
5#
6# This file is part of charm-helpers.
7#
8# charm-helpers is free software: you can redistribute it and/or modify
9# it under the terms of the GNU Lesser General Public License version 3 as
10# published by the Free Software Foundation.
11#
12# charm-helpers is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU Lesser General Public License for more details.
16#
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
19
20__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
21
22from charmhelpers.core.hookenv import (
23 log,
24 INFO
25)
26
27from subprocess import check_call, check_output
28import re
29
30
31def modprobe(module, persist=True):
32 """Load a kernel module and configure for auto-load on reboot."""
33 cmd = ['modprobe', module]
34
35 log('Loading kernel module %s' % module, level=INFO)
36
37 check_call(cmd)
38 if persist:
39 with open('/etc/modules', 'r+') as modules:
40 if module not in modules.read():
41 modules.write(module)
42
43
44def rmmod(module, force=False):
45 """Remove a module from the linux kernel"""
46 cmd = ['rmmod']
47 if force:
48 cmd.append('-f')
49 cmd.append(module)
50 log('Removing kernel module %s' % module, level=INFO)
51 return check_call(cmd)
52
53
54def lsmod():
55 """Shows what kernel modules are currently loaded"""
56 return check_output(['lsmod'],
57 universal_newlines=True)
58
59
60def is_module_loaded(module):
61 """Checks if a kernel module is already loaded"""
62 matches = re.findall('^%s[ ]+' % module, lsmod(), re.M)
63 return len(matches) > 0
64
65
66def update_initramfs(version='all'):
67 """Updates an initramfs image"""
68 return check_call(["update-initramfs", "-k", version, "-u"])
069
=== modified file 'lib/charmhelpers/core/services/__init__.py'
--- lib/charmhelpers/core/services/__init__.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/services/__init__.py 2016-07-12 15:38:59 +0000
@@ -1,2 +1,18 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1from .base import * # NOQA17from .base import * # NOQA
2from .helpers import * # NOQA18from .helpers import * # NOQA
319
=== modified file 'lib/charmhelpers/core/services/base.py'
--- lib/charmhelpers/core/services/base.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/services/base.py 2016-07-12 15:38:59 +0000
@@ -1,7 +1,23 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
2import re
3import json18import json
4from collections import Iterable19from inspect import getargspec
20from collections import Iterable, OrderedDict
521
6from charmhelpers.core import host22from charmhelpers.core import host
7from charmhelpers.core import hookenv23from charmhelpers.core import hookenv
@@ -103,7 +119,7 @@
103 """119 """
104 self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')120 self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
105 self._ready = None121 self._ready = None
106 self.services = {}122 self.services = OrderedDict()
107 for service in services or []:123 for service in services or []:
108 service_name = service['service']124 service_name = service['service']
109 self.services[service_name] = service125 self.services[service_name] = service
@@ -112,15 +128,18 @@
112 """128 """
113 Handle the current hook by doing The Right Thing with the registered services.129 Handle the current hook by doing The Right Thing with the registered services.
114 """130 """
115 hook_name = hookenv.hook_name()131 hookenv._run_atstart()
116 if hook_name == 'stop':132 try:
117 self.stop_services()133 hook_name = hookenv.hook_name()
118 else:134 if hook_name == 'stop':
119 self.provide_data()135 self.stop_services()
120 self.reconfigure_services()136 else:
121 cfg = hookenv.config()137 self.reconfigure_services()
122 if cfg.implicit_save:138 self.provide_data()
123 cfg.save()139 except SystemExit as x:
140 if x.code is None or x.code == 0:
141 hookenv._run_atexit()
142 hookenv._run_atexit()
124143
125 def provide_data(self):144 def provide_data(self):
126 """145 """
@@ -129,15 +148,36 @@
129 A provider must have a `name` attribute, which indicates which relation148 A provider must have a `name` attribute, which indicates which relation
130 to set data on, and a `provide_data()` method, which returns a dict of149 to set data on, and a `provide_data()` method, which returns a dict of
131 data to set.150 data to set.
151
152 The `provide_data()` method can optionally accept two parameters:
153
154 * ``remote_service`` The name of the remote service that the data will
155 be provided to. The `provide_data()` method will be called once
156 for each connected service (not unit). This allows the method to
157 tailor its data to the given service.
158 * ``service_ready`` Whether or not the service definition had all of
159 its requirements met, and thus the ``data_ready`` callbacks run.
160
161 Note that the ``provided_data`` methods are now called **after** the
162 ``data_ready`` callbacks are run. This gives the ``data_ready`` callbacks
163 a chance to generate any data necessary for the providing to the remote
164 services.
132 """165 """
133 hook_name = hookenv.hook_name()166 for service_name, service in self.services.items():
134 for service in self.services.values():167 service_ready = self.is_ready(service_name)
135 for provider in service.get('provided_data', []):168 for provider in service.get('provided_data', []):
136 if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name):169 for relid in hookenv.relation_ids(provider.name):
137 data = provider.provide_data()170 units = hookenv.related_units(relid)
138 _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data171 if not units:
139 if _ready:172 continue
140 hookenv.relation_set(None, data)173 remote_service = units[0].split('/')[0]
174 argspec = getargspec(provider.provide_data)
175 if len(argspec.args) > 1:
176 data = provider.provide_data(remote_service, service_ready)
177 else:
178 data = provider.provide_data()
179 if data:
180 hookenv.relation_set(relid, data)
141181
142 def reconfigure_services(self, *service_names):182 def reconfigure_services(self, *service_names):
143 """183 """
144184
=== modified file 'lib/charmhelpers/core/services/helpers.py'
--- lib/charmhelpers/core/services/helpers.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/services/helpers.py 2016-07-12 15:38:59 +0000
@@ -1,6 +1,24 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
2import yaml18import yaml
19
3from charmhelpers.core import hookenv20from charmhelpers.core import hookenv
21from charmhelpers.core import host
4from charmhelpers.core import templating22from charmhelpers.core import templating
523
6from charmhelpers.core.services.base import ManagerCallback24from charmhelpers.core.services.base import ManagerCallback
@@ -29,12 +47,14 @@
29 """47 """
30 name = None48 name = None
31 interface = None49 interface = None
32 required_keys = []
3350
34 def __init__(self, name=None, additional_required_keys=None):51 def __init__(self, name=None, additional_required_keys=None):
52 if not hasattr(self, 'required_keys'):
53 self.required_keys = []
54
35 if name is not None:55 if name is not None:
36 self.name = name56 self.name = name
37 if additional_required_keys is not None:57 if additional_required_keys:
38 self.required_keys.extend(additional_required_keys)58 self.required_keys.extend(additional_required_keys)
39 self.get_data()59 self.get_data()
4060
@@ -118,7 +138,10 @@
118 """138 """
119 name = 'db'139 name = 'db'
120 interface = 'mysql'140 interface = 'mysql'
121 required_keys = ['host', 'user', 'password', 'database']141
142 def __init__(self, *args, **kwargs):
143 self.required_keys = ['host', 'user', 'password', 'database']
144 RelationContext.__init__(self, *args, **kwargs)
122145
123146
124class HttpRelation(RelationContext):147class HttpRelation(RelationContext):
@@ -130,7 +153,10 @@
130 """153 """
131 name = 'website'154 name = 'website'
132 interface = 'http'155 interface = 'http'
133 required_keys = ['host', 'port']156
157 def __init__(self, *args, **kwargs):
158 self.required_keys = ['host', 'port']
159 RelationContext.__init__(self, *args, **kwargs)
134160
135 def provide_data(self):161 def provide_data(self):
136 return {162 return {
@@ -215,28 +241,51 @@
215 action.241 action.
216242
217 :param str source: The template source file, relative to243 :param str source: The template source file, relative to
218 `$CHARM_DIR/templates`244 `$CHARM_DIR/templates`
219245
220 :param str target: The target to write the rendered template to246 :param str target: The target to write the rendered template to (or None)
221 :param str owner: The owner of the rendered file247 :param str owner: The owner of the rendered file
222 :param str group: The group of the rendered file248 :param str group: The group of the rendered file
223 :param int perms: The permissions of the rendered file249 :param int perms: The permissions of the rendered file
250 :param partial on_change_action: functools partial to be executed when
251 rendered file changes
252 :param jinja2 loader template_loader: A jinja2 template loader
253
254 :return str: The rendered template
224 """255 """
225 def __init__(self, source, target,256 def __init__(self, source, target,
226 owner='root', group='root', perms=0o444):257 owner='root', group='root', perms=0o444,
258 on_change_action=None, template_loader=None):
227 self.source = source259 self.source = source
228 self.target = target260 self.target = target
229 self.owner = owner261 self.owner = owner
230 self.group = group262 self.group = group
231 self.perms = perms263 self.perms = perms
264 self.on_change_action = on_change_action
265 self.template_loader = template_loader
232266
233 def __call__(self, manager, service_name, event_name):267 def __call__(self, manager, service_name, event_name):
268 pre_checksum = ''
269 if self.on_change_action and os.path.isfile(self.target):
270 pre_checksum = host.file_hash(self.target)
234 service = manager.get_service(service_name)271 service = manager.get_service(service_name)
235 context = {}272 context = {'ctx': {}}
236 for ctx in service.get('required_data', []):273 for ctx in service.get('required_data', []):
237 context.update(ctx)274 context.update(ctx)
238 templating.render(self.source, self.target, context,275 context['ctx'].update(ctx)
239 self.owner, self.group, self.perms)276
277 result = templating.render(self.source, self.target, context,
278 self.owner, self.group, self.perms,
279 template_loader=self.template_loader)
280 if self.on_change_action:
281 if pre_checksum == host.file_hash(self.target):
282 hookenv.log(
283 'No change detected: {}'.format(self.target),
284 hookenv.DEBUG)
285 else:
286 self.on_change_action()
287
288 return result
240289
241290
242# Convenience aliases for templates291# Convenience aliases for templates
243292
=== added file 'lib/charmhelpers/core/strutils.py'
--- lib/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000
+++ lib/charmhelpers/core/strutils.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,72 @@
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3
4# Copyright 2014-2015 Canonical Limited.
5#
6# This file is part of charm-helpers.
7#
8# charm-helpers is free software: you can redistribute it and/or modify
9# it under the terms of the GNU Lesser General Public License version 3 as
10# published by the Free Software Foundation.
11#
12# charm-helpers is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU Lesser General Public License for more details.
16#
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
19
20import six
21import re
22
23
24def bool_from_string(value):
25 """Interpret string value as boolean.
26
27 Returns True if value translates to True otherwise False.
28 """
29 if isinstance(value, six.string_types):
30 value = six.text_type(value)
31 else:
32 msg = "Unable to interpret non-string value '%s' as boolean" % (value)
33 raise ValueError(msg)
34
35 value = value.strip().lower()
36
37 if value in ['y', 'yes', 'true', 't', 'on']:
38 return True
39 elif value in ['n', 'no', 'false', 'f', 'off']:
40 return False
41
42 msg = "Unable to interpret string value '%s' as boolean" % (value)
43 raise ValueError(msg)
44
45
46def bytes_from_string(value):
47 """Interpret human readable string value as bytes.
48
49 Returns int
50 """
51 BYTE_POWER = {
52 'K': 1,
53 'KB': 1,
54 'M': 2,
55 'MB': 2,
56 'G': 3,
57 'GB': 3,
58 'T': 4,
59 'TB': 4,
60 'P': 5,
61 'PB': 5,
62 }
63 if isinstance(value, six.string_types):
64 value = six.text_type(value)
65 else:
66 msg = "Unable to interpret non-string value '%s' as boolean" % (value)
67 raise ValueError(msg)
68 matches = re.match("([0-9]+)([a-zA-Z]+)", value)
69 if not matches:
70 msg = "Unable to interpret string value '%s' as bytes" % (value)
71 raise ValueError(msg)
72 return int(matches.group(1)) * (1024 ** BYTE_POWER[matches.group(2)])
073
=== modified file 'lib/charmhelpers/core/sysctl.py'
--- lib/charmhelpers/core/sysctl.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/sysctl.py 2016-07-12 15:38:59 +0000
@@ -1,7 +1,21 @@
1#!/usr/bin/env python1#!/usr/bin/env python
2# -*- coding: utf-8 -*-2# -*- coding: utf-8 -*-
33
4__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'4# Copyright 2014-2015 Canonical Limited.
5#
6# This file is part of charm-helpers.
7#
8# charm-helpers is free software: you can redistribute it and/or modify
9# it under the terms of the GNU Lesser General Public License version 3 as
10# published by the Free Software Foundation.
11#
12# charm-helpers is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU Lesser General Public License for more details.
16#
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
519
6import yaml20import yaml
721
@@ -10,25 +24,33 @@
10from charmhelpers.core.hookenv import (24from charmhelpers.core.hookenv import (
11 log,25 log,
12 DEBUG,26 DEBUG,
27 ERROR,
13)28)
1429
30__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
31
1532
16def create(sysctl_dict, sysctl_file):33def create(sysctl_dict, sysctl_file):
17 """Creates a sysctl.conf file from a YAML associative array34 """Creates a sysctl.conf file from a YAML associative array
1835
19 :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 }36 :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
20 :type sysctl_dict: dict37 :type sysctl_dict: str
21 :param sysctl_file: path to the sysctl file to be saved38 :param sysctl_file: path to the sysctl file to be saved
22 :type sysctl_file: str or unicode39 :type sysctl_file: str or unicode
23 :returns: None40 :returns: None
24 """41 """
25 sysctl_dict = yaml.load(sysctl_dict)42 try:
43 sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
44 except yaml.YAMLError:
45 log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
46 level=ERROR)
47 return
2648
27 with open(sysctl_file, "w") as fd:49 with open(sysctl_file, "w") as fd:
28 for key, value in sysctl_dict.items():50 for key, value in sysctl_dict_parsed.items():
29 fd.write("{}={}\n".format(key, value))51 fd.write("{}={}\n".format(key, value))
3052
31 log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict),53 log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
32 level=DEBUG)54 level=DEBUG)
3355
34 check_call(["sysctl", "-p", sysctl_file])56 check_call(["sysctl", "-p", sysctl_file])
3557
=== modified file 'lib/charmhelpers/core/templating.py'
--- lib/charmhelpers/core/templating.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/core/templating.py 2016-07-12 15:38:59 +0000
@@ -1,3 +1,19 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
218
3from charmhelpers.core import host19from charmhelpers.core import host
@@ -5,13 +21,14 @@
521
622
7def render(source, target, context, owner='root', group='root',23def render(source, target, context, owner='root', group='root',
8 perms=0o444, templates_dir=None):24 perms=0o444, templates_dir=None, encoding='UTF-8', template_loader=None):
9 """25 """
10 Render a template.26 Render a template.
1127
12 The `source` path, if not absolute, is relative to the `templates_dir`.28 The `source` path, if not absolute, is relative to the `templates_dir`.
1329
14 The `target` path should be absolute.30 The `target` path should be absolute. It can also be `None`, in which
31 case no file will be written.
1532
16 The context should be a dict containing the values to be replaced in the33 The context should be a dict containing the values to be replaced in the
17 template.34 template.
@@ -20,6 +37,9 @@
2037
21 If omitted, `templates_dir` defaults to the `templates` folder in the charm.38 If omitted, `templates_dir` defaults to the `templates` folder in the charm.
2239
40 The rendered template will be written to the file as well as being returned
41 as a string.
42
23 Note: Using this requires python-jinja2; if it is not installed, calling43 Note: Using this requires python-jinja2; if it is not installed, calling
24 this will attempt to use charmhelpers.fetch.apt_install to install it.44 this will attempt to use charmhelpers.fetch.apt_install to install it.
25 """45 """
@@ -36,17 +56,26 @@
36 apt_install('python-jinja2', fatal=True)56 apt_install('python-jinja2', fatal=True)
37 from jinja2 import FileSystemLoader, Environment, exceptions57 from jinja2 import FileSystemLoader, Environment, exceptions
3858
39 if templates_dir is None:59 if template_loader:
40 templates_dir = os.path.join(hookenv.charm_dir(), 'templates')60 template_env = Environment(loader=template_loader)
41 loader = Environment(loader=FileSystemLoader(templates_dir))61 else:
62 if templates_dir is None:
63 templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
64 template_env = Environment(loader=FileSystemLoader(templates_dir))
42 try:65 try:
43 source = source66 source = source
44 template = loader.get_template(source)67 template = template_env.get_template(source)
45 except exceptions.TemplateNotFound as e:68 except exceptions.TemplateNotFound as e:
46 hookenv.log('Could not load template %s from %s.' %69 hookenv.log('Could not load template %s from %s.' %
47 (source, templates_dir),70 (source, templates_dir),
48 level=hookenv.ERROR)71 level=hookenv.ERROR)
49 raise e72 raise e
50 content = template.render(context)73 content = template.render(context)
51 host.mkdir(os.path.dirname(target))74 if target is not None:
52 host.write_file(target, content, owner, group, perms)75 target_dir = os.path.dirname(target)
76 if not os.path.exists(target_dir):
77 # This is a terrible default directory permission, as the file
78 # or its siblings will often contain secrets.
79 host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
80 host.write_file(target, content.encode(encoding), owner, group, perms)
81 return content
5382
=== added file 'lib/charmhelpers/core/unitdata.py'
--- lib/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000
+++ lib/charmhelpers/core/unitdata.py 2016-07-12 15:38:59 +0000
@@ -0,0 +1,521 @@
1#!/usr/bin/env python
2# -*- coding: utf-8 -*-
3#
4# Copyright 2014-2015 Canonical Limited.
5#
6# This file is part of charm-helpers.
7#
8# charm-helpers is free software: you can redistribute it and/or modify
9# it under the terms of the GNU Lesser General Public License version 3 as
10# published by the Free Software Foundation.
11#
12# charm-helpers is distributed in the hope that it will be useful,
13# but WITHOUT ANY WARRANTY; without even the implied warranty of
14# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
15# GNU Lesser General Public License for more details.
16#
17# You should have received a copy of the GNU Lesser General Public License
18# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
19#
20#
21# Authors:
22# Kapil Thangavelu <kapil.foss@gmail.com>
23#
24"""
25Intro
26-----
27
28A simple way to store state in units. This provides a key value
29storage with support for versioned, transactional operation,
30and can calculate deltas from previous values to simplify unit logic
31when processing changes.
32
33
34Hook Integration
35----------------
36
37There are several extant frameworks for hook execution, including
38
39 - charmhelpers.core.hookenv.Hooks
40 - charmhelpers.core.services.ServiceManager
41
42The storage classes are framework agnostic, one simple integration is
43via the HookData contextmanager. It will record the current hook
44execution environment (including relation data, config data, etc.),
45setup a transaction and allow easy access to the changes from
46previously seen values. One consequence of the integration is the
47reservation of particular keys ('rels', 'unit', 'env', 'config',
48'charm_revisions') for their respective values.
49
50Here's a fully worked integration example using hookenv.Hooks::
51
52 from charmhelper.core import hookenv, unitdata
53
54 hook_data = unitdata.HookData()
55 db = unitdata.kv()
56 hooks = hookenv.Hooks()
57
58 @hooks.hook
59 def config_changed():
60 # Print all changes to configuration from previously seen
61 # values.
62 for changed, (prev, cur) in hook_data.conf.items():
63 print('config changed', changed,
64 'previous value', prev,
65 'current value', cur)
66
67 # Get some unit specific bookeeping
68 if not db.get('pkg_key'):
69 key = urllib.urlopen('https://example.com/pkg_key').read()
70 db.set('pkg_key', key)
71
72 # Directly access all charm config as a mapping.
73 conf = db.getrange('config', True)
74
75 # Directly access all relation data as a mapping
76 rels = db.getrange('rels', True)
77
78 if __name__ == '__main__':
79 with hook_data():
80 hook.execute()
81
82
83A more basic integration is via the hook_scope context manager which simply
84manages transaction scope (and records hook name, and timestamp)::
85
86 >>> from unitdata import kv
87 >>> db = kv()
88 >>> with db.hook_scope('install'):
89 ... # do work, in transactional scope.
90 ... db.set('x', 1)
91 >>> db.get('x')
92 1
93
94
95Usage
96-----
97
98Values are automatically json de/serialized to preserve basic typing
99and complex data struct capabilities (dicts, lists, ints, booleans, etc).
100
101Individual values can be manipulated via get/set::
102
103 >>> kv.set('y', True)
104 >>> kv.get('y')
105 True
106
107 # We can set complex values (dicts, lists) as a single key.
108 >>> kv.set('config', {'a': 1, 'b': True'})
109
110 # Also supports returning dictionaries as a record which
111 # provides attribute access.
112 >>> config = kv.get('config', record=True)
113 >>> config.b
114 True
115
116
117Groups of keys can be manipulated with update/getrange::
118
119 >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
120 >>> kv.getrange('gui.', strip=True)
121 {'z': 1, 'y': 2}
122
123When updating values, its very helpful to understand which values
124have actually changed and how have they changed. The storage
125provides a delta method to provide for this::
126
127 >>> data = {'debug': True, 'option': 2}
128 >>> delta = kv.delta(data, 'config.')
129 >>> delta.debug.previous
130 None
131 >>> delta.debug.current
132 True
133 >>> delta
134 {'debug': (None, True), 'option': (None, 2)}
135
136Note the delta method does not persist the actual change, it needs to
137be explicitly saved via 'update' method::
138
139 >>> kv.update(data, 'config.')
140
141Values modified in the context of a hook scope retain historical values
142associated to the hookname.
143
144 >>> with db.hook_scope('config-changed'):
145 ... db.set('x', 42)
146 >>> db.gethistory('x')
147 [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
148 (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
149
150"""
151
152import collections
153import contextlib
154import datetime
155import itertools
156import json
157import os
158import pprint
159import sqlite3
160import sys
161
162__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
163
164
165class Storage(object):
166 """Simple key value database for local unit state within charms.
167
168 Modifications are not persisted unless :meth:`flush` is called.
169
170 To support dicts, lists, integer, floats, and booleans values
171 are automatically json encoded/decoded.
172 """
173 def __init__(self, path=None):
174 self.db_path = path
175 if path is None:
176 if 'UNIT_STATE_DB' in os.environ:
177 self.db_path = os.environ['UNIT_STATE_DB']
178 else:
179 self.db_path = os.path.join(
180 os.environ.get('CHARM_DIR', ''), '.unit-state.db')
181 self.conn = sqlite3.connect('%s' % self.db_path)
182 self.cursor = self.conn.cursor()
183 self.revision = None
184 self._closed = False
185 self._init()
186
187 def close(self):
188 if self._closed:
189 return
190 self.flush(False)
191 self.cursor.close()
192 self.conn.close()
193 self._closed = True
194
195 def get(self, key, default=None, record=False):
196 self.cursor.execute('select data from kv where key=?', [key])
197 result = self.cursor.fetchone()
198 if not result:
199 return default
200 if record:
201 return Record(json.loads(result[0]))
202 return json.loads(result[0])
203
204 def getrange(self, key_prefix, strip=False):
205 """
206 Get a range of keys starting with a common prefix as a mapping of
207 keys to values.
208
209 :param str key_prefix: Common prefix among all keys
210 :param bool strip: Optionally strip the common prefix from the key
211 names in the returned dict
212 :return dict: A (possibly empty) dict of key-value mappings
213 """
214 self.cursor.execute("select key, data from kv where key like ?",
215 ['%s%%' % key_prefix])
216 result = self.cursor.fetchall()
217
218 if not result:
219 return {}
220 if not strip:
221 key_prefix = ''
222 return dict([
223 (k[len(key_prefix):], json.loads(v)) for k, v in result])
224
225 def update(self, mapping, prefix=""):
226 """
227 Set the values of multiple keys at once.
228
229 :param dict mapping: Mapping of keys to values
230 :param str prefix: Optional prefix to apply to all keys in `mapping`
231 before setting
232 """
233 for k, v in mapping.items():
234 self.set("%s%s" % (prefix, k), v)
235
236 def unset(self, key):
237 """
238 Remove a key from the database entirely.
239 """
240 self.cursor.execute('delete from kv where key=?', [key])
241 if self.revision and self.cursor.rowcount:
242 self.cursor.execute(
243 'insert into kv_revisions values (?, ?, ?)',
244 [key, self.revision, json.dumps('DELETED')])
245
246 def unsetrange(self, keys=None, prefix=""):
247 """
248 Remove a range of keys starting with a common prefix, from the database
249 entirely.
250
251 :param list keys: List of keys to remove.
252 :param str prefix: Optional prefix to apply to all keys in ``keys``
253 before removing.
254 """
255 if keys is not None:
256 keys = ['%s%s' % (prefix, key) for key in keys]
257 self.cursor.execute('delete from kv where key in (%s)' % ','.join(['?'] * len(keys)), keys)
258 if self.revision and self.cursor.rowcount:
259 self.cursor.execute(
260 'insert into kv_revisions values %s' % ','.join(['(?, ?, ?)'] * len(keys)),
261 list(itertools.chain.from_iterable((key, self.revision, json.dumps('DELETED')) for key in keys)))
262 else:
263 self.cursor.execute('delete from kv where key like ?',
264 ['%s%%' % prefix])
265 if self.revision and self.cursor.rowcount:
266 self.cursor.execute(
267 'insert into kv_revisions values (?, ?, ?)',
268 ['%s%%' % prefix, self.revision, json.dumps('DELETED')])
269
270 def set(self, key, value):
271 """
272 Set a value in the database.
273
274 :param str key: Key to set the value for
275 :param value: Any JSON-serializable value to be set
276 """
277 serialized = json.dumps(value)
278
279 self.cursor.execute('select data from kv where key=?', [key])
280 exists = self.cursor.fetchone()
281
282 # Skip mutations to the same value
283 if exists:
284 if exists[0] == serialized:
285 return value
286
287 if not exists:
288 self.cursor.execute(
289 'insert into kv (key, data) values (?, ?)',
290 (key, serialized))
291 else:
292 self.cursor.execute('''
293 update kv
294 set data = ?
295 where key = ?''', [serialized, key])
296
297 # Save
298 if not self.revision:
299 return value
300
301 self.cursor.execute(
302 'select 1 from kv_revisions where key=? and revision=?',
303 [key, self.revision])
304 exists = self.cursor.fetchone()
305
306 if not exists:
307 self.cursor.execute(
308 '''insert into kv_revisions (
309 revision, key, data) values (?, ?, ?)''',
310 (self.revision, key, serialized))
311 else:
312 self.cursor.execute(
313 '''
314 update kv_revisions
315 set data = ?
316 where key = ?
317 and revision = ?''',
318 [serialized, key, self.revision])
319
320 return value
321
322 def delta(self, mapping, prefix):
323 """
324 return a delta containing values that have changed.
325 """
326 previous = self.getrange(prefix, strip=True)
327 if not previous:
328 pk = set()
329 else:
330 pk = set(previous.keys())
331 ck = set(mapping.keys())
332 delta = DeltaSet()
333
334 # added
335 for k in ck.difference(pk):
336 delta[k] = Delta(None, mapping[k])
337
338 # removed
339 for k in pk.difference(ck):
340 delta[k] = Delta(previous[k], None)
341
342 # changed
343 for k in pk.intersection(ck):
344 c = mapping[k]
345 p = previous[k]
346 if c != p:
347 delta[k] = Delta(p, c)
348
349 return delta
350
351 @contextlib.contextmanager
352 def hook_scope(self, name=""):
353 """Scope all future interactions to the current hook execution
354 revision."""
355 assert not self.revision
356 self.cursor.execute(
357 'insert into hooks (hook, date) values (?, ?)',
358 (name or sys.argv[0],
359 datetime.datetime.utcnow().isoformat()))
360 self.revision = self.cursor.lastrowid
361 try:
362 yield self.revision
363 self.revision = None
364 except:
365 self.flush(False)
366 self.revision = None
367 raise
368 else:
369 self.flush()
370
371 def flush(self, save=True):
372 if save:
373 self.conn.commit()
374 elif self._closed:
375 return
376 else:
377 self.conn.rollback()
378
379 def _init(self):
380 self.cursor.execute('''
381 create table if not exists kv (
382 key text,
383 data text,
384 primary key (key)
385 )''')
386 self.cursor.execute('''
387 create table if not exists kv_revisions (
388 key text,
389 revision integer,
390 data text,
391 primary key (key, revision)
392 )''')
393 self.cursor.execute('''
394 create table if not exists hooks (
395 version integer primary key autoincrement,
396 hook text,
397 date text
398 )''')
399 self.conn.commit()
400
401 def gethistory(self, key, deserialize=False):
402 self.cursor.execute(
403 '''
404 select kv.revision, kv.key, kv.data, h.hook, h.date
405 from kv_revisions kv,
406 hooks h
407 where kv.key=?
408 and kv.revision = h.version
409 ''', [key])
410 if deserialize is False:
411 return self.cursor.fetchall()
412 return map(_parse_history, self.cursor.fetchall())
413
414 def debug(self, fh=sys.stderr):
415 self.cursor.execute('select * from kv')
416 pprint.pprint(self.cursor.fetchall(), stream=fh)
417 self.cursor.execute('select * from kv_revisions')
418 pprint.pprint(self.cursor.fetchall(), stream=fh)
419
420
421def _parse_history(d):
422 return (d[0], d[1], json.loads(d[2]), d[3],
423 datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
424
425
426class HookData(object):
427 """Simple integration for existing hook exec frameworks.
428
429 Records all unit information, and stores deltas for processing
430 by the hook.
431
432 Sample::
433
434 from charmhelper.core import hookenv, unitdata
435
436 changes = unitdata.HookData()
437 db = unitdata.kv()
438 hooks = hookenv.Hooks()
439
440 @hooks.hook
441 def config_changed():
442 # View all changes to configuration
443 for changed, (prev, cur) in changes.conf.items():
444 print('config changed', changed,
445 'previous value', prev,
446 'current value', cur)
447
448 # Get some unit specific bookeeping
449 if not db.get('pkg_key'):
450 key = urllib.urlopen('https://example.com/pkg_key').read()
451 db.set('pkg_key', key)
452
453 if __name__ == '__main__':
454 with changes():
455 hook.execute()
456
457 """
458 def __init__(self):
459 self.kv = kv()
460 self.conf = None
461 self.rels = None
462
463 @contextlib.contextmanager
464 def __call__(self):
465 from charmhelpers.core import hookenv
466 hook_name = hookenv.hook_name()
467
468 with self.kv.hook_scope(hook_name):
469 self._record_charm_version(hookenv.charm_dir())
470 delta_config, delta_relation = self._record_hook(hookenv)
471 yield self.kv, delta_config, delta_relation
472
473 def _record_charm_version(self, charm_dir):
474 # Record revisions.. charm revisions are meaningless
475 # to charm authors as they don't control the revision.
476 # so logic dependnent on revision is not particularly
477 # useful, however it is useful for debugging analysis.
478 charm_rev = open(
479 os.path.join(charm_dir, 'revision')).read().strip()
480 charm_rev = charm_rev or '0'
481 revs = self.kv.get('charm_revisions', [])
482 if charm_rev not in revs:
483 revs.append(charm_rev.strip() or '0')
484 self.kv.set('charm_revisions', revs)
485
486 def _record_hook(self, hookenv):
487 data = hookenv.execution_environment()
488 self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
489 self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
490 self.kv.set('env', dict(data['env']))
491 self.kv.set('unit', data['unit'])
492 self.kv.set('relid', data.get('relid'))
493 return conf_delta, rels_delta
494
495
496class Record(dict):
497
498 __slots__ = ()
499
500 def __getattr__(self, k):
501 if k in self:
502 return self[k]
503 raise AttributeError(k)
504
505
506class DeltaSet(Record):
507
508 __slots__ = ()
509
510
511Delta = collections.namedtuple('Delta', ['previous', 'current'])
512
513
514_KV = None
515
516
517def kv():
518 global _KV
519 if _KV is None:
520 _KV = Storage()
521 return _KV
0522
=== modified file 'lib/charmhelpers/fetch/__init__.py'
--- lib/charmhelpers/fetch/__init__.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/fetch/__init__.py 2016-07-12 15:38:59 +0000
@@ -1,3 +1,19 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import importlib17import importlib
2from tempfile import NamedTemporaryFile18from tempfile import NamedTemporaryFile
3import time19import time
@@ -64,9 +80,40 @@
64 'trusty-juno/updates': 'trusty-updates/juno',80 'trusty-juno/updates': 'trusty-updates/juno',
65 'trusty-updates/juno': 'trusty-updates/juno',81 'trusty-updates/juno': 'trusty-updates/juno',
66 'juno/proposed': 'trusty-proposed/juno',82 'juno/proposed': 'trusty-proposed/juno',
67 'juno/proposed': 'trusty-proposed/juno',
68 'trusty-juno/proposed': 'trusty-proposed/juno',83 'trusty-juno/proposed': 'trusty-proposed/juno',
69 'trusty-proposed/juno': 'trusty-proposed/juno',84 'trusty-proposed/juno': 'trusty-proposed/juno',
85 # Kilo
86 'kilo': 'trusty-updates/kilo',
87 'trusty-kilo': 'trusty-updates/kilo',
88 'trusty-kilo/updates': 'trusty-updates/kilo',
89 'trusty-updates/kilo': 'trusty-updates/kilo',
90 'kilo/proposed': 'trusty-proposed/kilo',
91 'trusty-kilo/proposed': 'trusty-proposed/kilo',
92 'trusty-proposed/kilo': 'trusty-proposed/kilo',
93 # Liberty
94 'liberty': 'trusty-updates/liberty',
95 'trusty-liberty': 'trusty-updates/liberty',
96 'trusty-liberty/updates': 'trusty-updates/liberty',
97 'trusty-updates/liberty': 'trusty-updates/liberty',
98 'liberty/proposed': 'trusty-proposed/liberty',
99 'trusty-liberty/proposed': 'trusty-proposed/liberty',
100 'trusty-proposed/liberty': 'trusty-proposed/liberty',
101 # Mitaka
102 'mitaka': 'trusty-updates/mitaka',
103 'trusty-mitaka': 'trusty-updates/mitaka',
104 'trusty-mitaka/updates': 'trusty-updates/mitaka',
105 'trusty-updates/mitaka': 'trusty-updates/mitaka',
106 'mitaka/proposed': 'trusty-proposed/mitaka',
107 'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
108 'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
109 # Newton
110 'newton': 'xenial-updates/newton',
111 'xenial-newton': 'xenial-updates/newton',
112 'xenial-newton/updates': 'xenial-updates/newton',
113 'xenial-updates/newton': 'xenial-updates/newton',
114 'newton/proposed': 'xenial-proposed/newton',
115 'xenial-newton/proposed': 'xenial-proposed/newton',
116 'xenial-proposed/newton': 'xenial-proposed/newton',
70}117}
71118
72# The order of this list is very important. Handlers should be listed in from119# The order of this list is very important. Handlers should be listed in from
@@ -135,7 +182,7 @@
135182
136def apt_cache(in_memory=True):183def apt_cache(in_memory=True):
137 """Build and return an apt cache"""184 """Build and return an apt cache"""
138 import apt_pkg185 from apt import apt_pkg
139 apt_pkg.init()186 apt_pkg.init()
140 if in_memory:187 if in_memory:
141 apt_pkg.config.set("Dir::Cache::pkgcache", "")188 apt_pkg.config.set("Dir::Cache::pkgcache", "")
@@ -192,19 +239,27 @@
192 _run_apt_command(cmd, fatal)239 _run_apt_command(cmd, fatal)
193240
194241
242def apt_mark(packages, mark, fatal=False):
243 """Flag one or more packages using apt-mark"""
244 log("Marking {} as {}".format(packages, mark))
245 cmd = ['apt-mark', mark]
246 if isinstance(packages, six.string_types):
247 cmd.append(packages)
248 else:
249 cmd.extend(packages)
250
251 if fatal:
252 subprocess.check_call(cmd, universal_newlines=True)
253 else:
254 subprocess.call(cmd, universal_newlines=True)
255
256
195def apt_hold(packages, fatal=False):257def apt_hold(packages, fatal=False):
196 """Hold one or more packages"""258 return apt_mark(packages, 'hold', fatal=fatal)
197 cmd = ['apt-mark', 'hold']259
198 if isinstance(packages, six.string_types):260
199 cmd.append(packages)261def apt_unhold(packages, fatal=False):
200 else:262 return apt_mark(packages, 'unhold', fatal=fatal)
201 cmd.extend(packages)
202 log("Holding {}".format(packages))
203
204 if fatal:
205 subprocess.check_call(cmd)
206 else:
207 subprocess.call(cmd)
208263
209264
210def add_source(source, key=None):265def add_source(source, key=None):
@@ -343,15 +398,13 @@
343 # We ONLY check for True here because can_handle may return a string398 # We ONLY check for True here because can_handle may return a string
344 # explaining why it can't handle a given source.399 # explaining why it can't handle a given source.
345 handlers = [h for h in plugins() if h.can_handle(source) is True]400 handlers = [h for h in plugins() if h.can_handle(source) is True]
346 installed_to = None
347 for handler in handlers:401 for handler in handlers:
348 try:402 try:
349 installed_to = handler.install(source, *args, **kwargs)403 return handler.install(source, *args, **kwargs)
350 except UnhandledSource:404 except UnhandledSource as e:
351 pass405 log('Install source attempt unsuccessful: {}'.format(e),
352 if not installed_to:406 level='WARNING')
353 raise UnhandledSource("No handler found for source {}".format(source))407 raise UnhandledSource("No handler found for source {}".format(source))
354 return installed_to
355408
356409
357def install_from_config(config_var_name):410def install_from_config(config_var_name):
@@ -371,7 +424,7 @@
371 importlib.import_module(package),424 importlib.import_module(package),
372 classname)425 classname)
373 plugin_list.append(handler_class())426 plugin_list.append(handler_class())
374 except (ImportError, AttributeError):427 except NotImplementedError:
375 # Skip missing plugins so that they can be ommitted from428 # Skip missing plugins so that they can be ommitted from
376 # installation if desired429 # installation if desired
377 log("FetchHandler {} not found, skipping plugin".format(430 log("FetchHandler {} not found, skipping plugin".format(
378431
=== modified file 'lib/charmhelpers/fetch/archiveurl.py'
--- lib/charmhelpers/fetch/archiveurl.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/fetch/archiveurl.py 2016-07-12 15:38:59 +0000
@@ -1,7 +1,33 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
2import hashlib18import hashlib
3import re19import re
420
21from charmhelpers.fetch import (
22 BaseFetchHandler,
23 UnhandledSource
24)
25from charmhelpers.payload.archive import (
26 get_archive_handler,
27 extract,
28)
29from charmhelpers.core.host import mkdir, check_hash
30
5import six31import six
6if six.PY3:32if six.PY3:
7 from urllib.request import (33 from urllib.request import (
@@ -19,16 +45,6 @@
19 )45 )
20 from urlparse import urlparse, urlunparse, parse_qs46 from urlparse import urlparse, urlunparse, parse_qs
2147
22from charmhelpers.fetch import (
23 BaseFetchHandler,
24 UnhandledSource
25)
26from charmhelpers.payload.archive import (
27 get_archive_handler,
28 extract,
29)
30from charmhelpers.core.host import mkdir, check_hash
31
3248
33def splituser(host):49def splituser(host):
34 '''urllib.splituser(), but six's support of this seems broken'''50 '''urllib.splituser(), but six's support of this seems broken'''
@@ -61,6 +77,8 @@
61 def can_handle(self, source):77 def can_handle(self, source):
62 url_parts = self.parse_url(source)78 url_parts = self.parse_url(source)
63 if url_parts.scheme not in ('http', 'https', 'ftp', 'file'):79 if url_parts.scheme not in ('http', 'https', 'ftp', 'file'):
80 # XXX: Why is this returning a boolean and a string? It's
81 # doomed to fail since "bool(can_handle('foo://'))" will be True.
64 return "Wrong source type"82 return "Wrong source type"
65 if get_archive_handler(self.base_url(source)):83 if get_archive_handler(self.base_url(source)):
66 return True84 return True
@@ -90,7 +108,7 @@
90 install_opener(opener)108 install_opener(opener)
91 response = urlopen(source)109 response = urlopen(source)
92 try:110 try:
93 with open(dest, 'w') as dest_file:111 with open(dest, 'wb') as dest_file:
94 dest_file.write(response.read())112 dest_file.write(response.read())
95 except Exception as e:113 except Exception as e:
96 if os.path.isfile(dest):114 if os.path.isfile(dest):
@@ -139,7 +157,11 @@
139 else:157 else:
140 algorithms = hashlib.algorithms_available158 algorithms = hashlib.algorithms_available
141 if key in algorithms:159 if key in algorithms:
142 check_hash(dld_file, value, key)160 if len(value) != 1:
161 raise TypeError(
162 "Expected 1 hash value, not %d" % len(value))
163 expected = value[0]
164 check_hash(dld_file, expected, key)
143 if checksum:165 if checksum:
144 check_hash(dld_file, checksum, hash_type)166 check_hash(dld_file, checksum, hash_type)
145 return extract(dld_file, dest)167 return extract(dld_file, dest)
146168
=== modified file 'lib/charmhelpers/fetch/bzrurl.py'
--- lib/charmhelpers/fetch/bzrurl.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/fetch/bzrurl.py 2016-07-12 15:38:59 +0000
@@ -1,54 +1,77 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
18from subprocess import check_call
2from charmhelpers.fetch import (19from charmhelpers.fetch import (
3 BaseFetchHandler,20 BaseFetchHandler,
4 UnhandledSource21 UnhandledSource,
22 filter_installed_packages,
23 apt_install,
5)24)
6from charmhelpers.core.host import mkdir25from charmhelpers.core.host import mkdir
726
8import six
9if six.PY3:
10 raise ImportError('bzrlib does not support Python3')
1127
12try:28if filter_installed_packages(['bzr']) != []:
13 from bzrlib.branch import Branch29 apt_install(['bzr'])
14except ImportError:30 if filter_installed_packages(['bzr']) != []:
15 from charmhelpers.fetch import apt_install31 raise NotImplementedError('Unable to install bzr')
16 apt_install("python-bzrlib")
17 from bzrlib.branch import Branch
1832
1933
20class BzrUrlFetchHandler(BaseFetchHandler):34class BzrUrlFetchHandler(BaseFetchHandler):
21 """Handler for bazaar branches via generic and lp URLs"""35 """Handler for bazaar branches via generic and lp URLs"""
22 def can_handle(self, source):36 def can_handle(self, source):
23 url_parts = self.parse_url(source)37 url_parts = self.parse_url(source)
24 if url_parts.scheme not in ('bzr+ssh', 'lp'):38 if url_parts.scheme not in ('bzr+ssh', 'lp', ''):
25 return False39 return False
40 elif not url_parts.scheme:
41 return os.path.exists(os.path.join(source, '.bzr'))
26 else:42 else:
27 return True43 return True
2844
29 def branch(self, source, dest):45 def branch(self, source, dest, revno=None):
30 url_parts = self.parse_url(source)
31 # If we use lp:branchname scheme we need to load plugins
32 if not self.can_handle(source):46 if not self.can_handle(source):
33 raise UnhandledSource("Cannot handle {}".format(source))47 raise UnhandledSource("Cannot handle {}".format(source))
34 if url_parts.scheme == "lp":48 cmd_opts = []
35 from bzrlib.plugin import load_plugins49 if revno:
36 load_plugins()50 cmd_opts += ['-r', str(revno)]
37 try:51 if os.path.exists(dest):
38 remote_branch = Branch.open(source)52 cmd = ['bzr', 'pull']
39 remote_branch.bzrdir.sprout(dest).open_branch()53 cmd += cmd_opts
40 except Exception as e:54 cmd += ['--overwrite', '-d', dest, source]
41 raise e55 else:
56 cmd = ['bzr', 'branch']
57 cmd += cmd_opts
58 cmd += [source, dest]
59 check_call(cmd)
4260
43 def install(self, source):61 def install(self, source, dest=None, revno=None):
44 url_parts = self.parse_url(source)62 url_parts = self.parse_url(source)
45 branch_name = url_parts.path.strip("/").split("/")[-1]63 branch_name = url_parts.path.strip("/").split("/")[-1]
46 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",64 if dest:
47 branch_name)65 dest_dir = os.path.join(dest, branch_name)
48 if not os.path.exists(dest_dir):66 else:
49 mkdir(dest_dir, perms=0o755)67 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
68 branch_name)
69
70 if dest and not os.path.exists(dest):
71 mkdir(dest, perms=0o755)
72
50 try:73 try:
51 self.branch(source, dest_dir)74 self.branch(source, dest_dir, revno)
52 except OSError as e:75 except OSError as e:
53 raise UnhandledSource(e.strerror)76 raise UnhandledSource(e.strerror)
54 return dest_dir77 return dest_dir
5578
=== modified file 'lib/charmhelpers/fetch/giturl.py'
--- lib/charmhelpers/fetch/giturl.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/fetch/giturl.py 2016-07-12 15:38:59 +0000
@@ -1,20 +1,32 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
18from subprocess import check_call, CalledProcessError
2from charmhelpers.fetch import (19from charmhelpers.fetch import (
3 BaseFetchHandler,20 BaseFetchHandler,
4 UnhandledSource21 UnhandledSource,
22 filter_installed_packages,
23 apt_install,
5)24)
6from charmhelpers.core.host import mkdir25
726if filter_installed_packages(['git']) != []:
8import six27 apt_install(['git'])
9if six.PY3:28 if filter_installed_packages(['git']) != []:
10 raise ImportError('GitPython does not support Python 3')29 raise NotImplementedError('Unable to install git')
11
12try:
13 from git import Repo
14except ImportError:
15 from charmhelpers.fetch import apt_install
16 apt_install("python-git")
17 from git import Repo
1830
1931
20class GitUrlFetchHandler(BaseFetchHandler):32class GitUrlFetchHandler(BaseFetchHandler):
@@ -22,27 +34,37 @@
22 def can_handle(self, source):34 def can_handle(self, source):
23 url_parts = self.parse_url(source)35 url_parts = self.parse_url(source)
24 # TODO (mattyw) no support for ssh git@ yet36 # TODO (mattyw) no support for ssh git@ yet
25 if url_parts.scheme not in ('http', 'https', 'git'):37 if url_parts.scheme not in ('http', 'https', 'git', ''):
26 return False38 return False
39 elif not url_parts.scheme:
40 return os.path.exists(os.path.join(source, '.git'))
27 else:41 else:
28 return True42 return True
2943
30 def clone(self, source, dest, branch):44 def clone(self, source, dest, branch="master", depth=None):
31 if not self.can_handle(source):45 if not self.can_handle(source):
32 raise UnhandledSource("Cannot handle {}".format(source))46 raise UnhandledSource("Cannot handle {}".format(source))
3347
34 repo = Repo.clone_from(source, dest)48 if os.path.exists(dest):
35 repo.git.checkout(branch)49 cmd = ['git', '-C', dest, 'pull', source, branch]
50 else:
51 cmd = ['git', 'clone', source, dest, '--branch', branch]
52 if depth:
53 cmd.extend(['--depth', depth])
54 check_call(cmd)
3655
37 def install(self, source, branch="master"):56 def install(self, source, branch="master", dest=None, depth=None):
38 url_parts = self.parse_url(source)57 url_parts = self.parse_url(source)
39 branch_name = url_parts.path.strip("/").split("/")[-1]58 branch_name = url_parts.path.strip("/").split("/")[-1]
40 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",59 if dest:
41 branch_name)60 dest_dir = os.path.join(dest, branch_name)
42 if not os.path.exists(dest_dir):61 else:
43 mkdir(dest_dir, perms=0o755)62 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
63 branch_name)
44 try:64 try:
45 self.clone(source, dest_dir, branch)65 self.clone(source, dest_dir, branch, depth)
66 except CalledProcessError as e:
67 raise UnhandledSource(e)
46 except OSError as e:68 except OSError as e:
47 raise UnhandledSource(e.strerror)69 raise UnhandledSource(e.strerror)
48 return dest_dir70 return dest_dir
4971
=== modified file 'lib/charmhelpers/payload/__init__.py'
--- lib/charmhelpers/payload/__init__.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/payload/__init__.py 2016-07-12 15:38:59 +0000
@@ -1,1 +1,17 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1"Tools for working with files injected into a charm just before deployment."17"Tools for working with files injected into a charm just before deployment."
218
=== modified file 'lib/charmhelpers/payload/archive.py'
--- lib/charmhelpers/payload/archive.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/payload/archive.py 2016-07-12 15:38:59 +0000
@@ -1,3 +1,19 @@
1# Copyright 2014-2015 Canonical Limited.
2#
3# This file is part of charm-helpers.
4#
5# charm-helpers is free software: you can redistribute it and/or modify
6# it under the terms of the GNU Lesser General Public License version 3 as
7# published by the Free Software Foundation.
8#
9# charm-helpers is distributed in the hope that it will be useful,
10# but WITHOUT ANY WARRANTY; without even the implied warranty of
11# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12# GNU Lesser General Public License for more details.
13#
14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
16
1import os17import os
2import tarfile18import tarfile
3import zipfile19import zipfile
420
=== modified file 'lib/charmhelpers/payload/execd.py'
--- lib/charmhelpers/payload/execd.py 2014-12-02 19:35:26 +0000
+++ lib/charmhelpers/payload/execd.py 2016-07-12 15:38:59 +0000
@@ -1,5 +1,21 @@
1#!/usr/bin/env python1#!/usr/bin/env python
22
3# Copyright 2014-2015 Canonical Limited.
4#
5# This file is part of charm-helpers.
6#
7# charm-helpers is free software: you can redistribute it and/or modify
8# it under the terms of the GNU Lesser General Public License version 3 as
9# published by the Free Software Foundation.
10#
11# charm-helpers is distributed in the hope that it will be useful,
12# but WITHOUT ANY WARRANTY; without even the implied warranty of
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14# GNU Lesser General Public License for more details.
15#
16# You should have received a copy of the GNU Lesser General Public License
17# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
18
3import os19import os
4import sys20import sys
5import subprocess21import subprocess
622
=== modified file 'metadata.yaml'
--- metadata.yaml 2014-11-12 21:43:04 +0000
+++ metadata.yaml 2016-07-12 15:38:59 +0000
@@ -1,6 +1,7 @@
1name: mariadb1name: mariadb
2summary: A Charm to install MariaDB2summary: A Charm to install MariaDB
3maintainer: Daniel Bartholomew <dbart@mariadb.com>3maintainer: Daniel Bartholomew <dbart@mariadb.com>
4series: [trusty, xenial]
4description: |5description: |
5 MariaDB is an open source database server. It can be used as the backing6 MariaDB is an open source database server. It can be used as the backing
6 database for web, business, and other applications and application servers.7 database for web, business, and other applications and application servers.
78
=== modified file 'revision'
--- revision 2016-05-31 20:38:01 +0000
+++ revision 2016-07-12 15:38:59 +0000
@@ -1,1 +1,1 @@
1312
22
=== modified file 'scripts/charm_helpers_sync.py'
--- scripts/charm_helpers_sync.py 2014-12-02 19:35:26 +0000
+++ scripts/charm_helpers_sync.py 2016-07-12 15:38:59 +0000
@@ -1,5 +1,20 @@
1#!/usr/bin/env python1#!/usr/bin/python
2# Copyright 2013 Canonical Ltd.2
3# Copyright 2014-2015 Canonical Limited.
4#
5# This file is part of charm-helpers.
6#
7# charm-helpers is free software: you can redistribute it and/or modify
8# it under the terms of the GNU Lesser General Public License version 3 as
9# published by the Free Software Foundation.
10#
11# charm-helpers is distributed in the hope that it will be useful,
12# but WITHOUT ANY WARRANTY; without even the implied warranty of
13# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14# GNU Lesser General Public License for more details.
15#
16# You should have received a copy of the GNU Lesser General Public License
17# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
318
4# Authors:19# Authors:
5# Adam Gandelman <adamg@ubuntu.com>20# Adam Gandelman <adamg@ubuntu.com>
@@ -12,9 +27,10 @@
12import sys27import sys
13import tempfile28import tempfile
14import yaml29import yaml
15
16from fnmatch import fnmatch30from fnmatch import fnmatch
1731
32import six
33
18CHARM_HELPERS_BRANCH = 'lp:charm-helpers'34CHARM_HELPERS_BRANCH = 'lp:charm-helpers'
1935
2036
@@ -28,7 +44,7 @@
28def clone_helpers(work_dir, branch):44def clone_helpers(work_dir, branch):
29 dest = os.path.join(work_dir, 'charm-helpers')45 dest = os.path.join(work_dir, 'charm-helpers')
30 logging.info('Checking out %s to %s.' % (branch, dest))46 logging.info('Checking out %s to %s.' % (branch, dest))
31 cmd = ['bzr', 'branch', branch, dest]47 cmd = ['bzr', 'checkout', '--lightweight', branch, dest]
32 subprocess.check_call(cmd)48 subprocess.check_call(cmd)
33 return dest49 return dest
3450
@@ -119,6 +135,20 @@
119135
120136
121def sync(src, dest, module, opts=None):137def sync(src, dest, module, opts=None):
138
139 # Sync charmhelpers/__init__.py for bootstrap code.
140 sync_pyfile(_src_path(src, '__init__'), dest)
141
142 # Sync other __init__.py files in the path leading to module.
143 m = []
144 steps = module.split('.')[:-1]
145 while steps:
146 m.append(steps.pop(0))
147 init = '.'.join(m + ['__init__'])
148 sync_pyfile(_src_path(src, init),
149 os.path.dirname(_dest_path(dest, init)))
150
151 # Sync the module, or maybe a .py file.
122 if os.path.isdir(_src_path(src, module)):152 if os.path.isdir(_src_path(src, module)):
123 sync_directory(_src_path(src, module), _dest_path(dest, module), opts)153 sync_directory(_src_path(src, module), _dest_path(dest, module), opts)
124 elif _is_pyfile(_src_path(src, module)):154 elif _is_pyfile(_src_path(src, module)):
@@ -137,7 +167,7 @@
137167
138def extract_options(inc, global_options=None):168def extract_options(inc, global_options=None):
139 global_options = global_options or []169 global_options = global_options or []
140 if global_options and isinstance(global_options, basestring):170 if global_options and isinstance(global_options, six.string_types):
141 global_options = [global_options]171 global_options = [global_options]
142 if '|' not in inc:172 if '|' not in inc:
143 return (inc, global_options)173 return (inc, global_options)
@@ -147,7 +177,7 @@
147177
148def sync_helpers(include, src, dest, options=None):178def sync_helpers(include, src, dest, options=None):
149 if not os.path.isdir(dest):179 if not os.path.isdir(dest):
150 os.mkdir(dest)180 os.makedirs(dest)
151181
152 global_options = parse_sync_options(options)182 global_options = parse_sync_options(options)
153183
@@ -157,7 +187,7 @@
157 sync(src, dest, inc, opts)187 sync(src, dest, inc, opts)
158 elif isinstance(inc, dict):188 elif isinstance(inc, dict):
159 # could also do nested dicts here.189 # could also do nested dicts here.
160 for k, v in inc.iteritems():190 for k, v in six.iteritems(inc):
161 if isinstance(v, list):191 if isinstance(v, list):
162 for m in v:192 for m in v:
163 inc, opts = extract_options(m, global_options)193 inc, opts = extract_options(m, global_options)
@@ -215,7 +245,7 @@
215 checkout = clone_helpers(tmpd, config['branch'])245 checkout = clone_helpers(tmpd, config['branch'])
216 sync_helpers(config['include'], checkout, config['destination'],246 sync_helpers(config['include'], checkout, config['destination'],
217 options=sync_options)247 options=sync_options)
218 except Exception, e:248 except Exception as e:
219 logging.error("Could not sync: %s" % e)249 logging.error("Could not sync: %s" % e)
220 raise e250 raise e
221 finally:251 finally:
222252
=== modified file 'tests/10-deploy-and-upgrade'
--- tests/10-deploy-and-upgrade 2015-11-03 22:05:55 +0000
+++ tests/10-deploy-and-upgrade 2016-07-12 15:38:59 +0000
@@ -11,17 +11,14 @@
11 cls.deployment = amulet.Deployment(series='trusty')11 cls.deployment = amulet.Deployment(series='trusty')
1212
13 mw_config = { 'name': 'MariaDB Test'}13 mw_config = { 'name': 'MariaDB Test'}
14 maria_config = { 'enterprise-eula': True, 'token': 'a202-wg2j'}14
15 cls.deployment.add('mariadb')15 cls.deployment.add('mariadb')
16 cls.deployment.add('mediawiki')16 cls.deployment.add('mediawiki')
17 cls.deployment.configure('mediawiki', mw_config)17 cls.deployment.configure('mediawiki', mw_config)
18 cls.deployment.configure('mariadb', maria_config)
19 cls.deployment.relate('mediawiki:db', 'mariadb:db')18 cls.deployment.relate('mediawiki:db', 'mariadb:db')
20 cls.deployment.expose('mediawiki')19 cls.deployment.expose('mediawiki')
2120
2221
23
24
25 try:22 try:
26 cls.deployment.setup(timeout=1200)23 cls.deployment.setup(timeout=1200)
27 cls.deployment.sentry.wait()24 cls.deployment.sentry.wait()
@@ -58,24 +55,7 @@
58 response = requests.get(wiki_url)55 response = requests.get(wiki_url)
59 response.raise_for_status()56 response.raise_for_status()
6057
61'''58
62 def test_enterprise_eval(self):
63 self.deployment.configure('mariadb', {'enterprise-eula': True,
64 'token': 'a202-wg2j'})
65
66 # Ensure the bintar was relocated
67 dbunit = self.deployment.sentry.unit['mariadb/0']
68
69 try:
70 dbunit.directory_stat('/usr/local/mysql')
71 amulet.raise_status(amulet.SKIP, 'bintar directory found, uncertain results ahead')
72 except:
73 # this is what we want to happen
74 pass
75
76 # re-run the test after in-place upgrade
77 self.test_credentials()
78'''
7959
80if __name__ == '__main__':60if __name__ == '__main__':
81 unittest.main()61 unittest.main()