Merge lp:~petevg/charms/trusty/mariadb/fixes-for-xenial into lp:~dbart/charms/trusty/mariadb/trunk

Proposed by Pete Vander Giessen on 2016-07-12
Status: Needs review
Proposed branch: lp:~petevg/charms/trusty/mariadb/fixes-for-xenial
Merge into: lp:~dbart/charms/trusty/mariadb/trunk
Diff against target: 3910 lines (+2438/-419)
33 files modified
README.md (+39/-50)
config.yaml (+10/-35)
hooks/common.py (+2/-2)
hooks/config-changed (+25/-46)
hooks/install (+2/-2)
hooks/start (+2/-1)
lib/charmhelpers/__init__.py (+38/-0)
lib/charmhelpers/core/__init__.py (+15/-0)
lib/charmhelpers/core/decorators.py (+57/-0)
lib/charmhelpers/core/files.py (+45/-0)
lib/charmhelpers/core/fstab.py (+19/-3)
lib/charmhelpers/core/hookenv.py (+516/-47)
lib/charmhelpers/core/host.py (+442/-74)
lib/charmhelpers/core/hugepage.py (+71/-0)
lib/charmhelpers/core/kernel.py (+68/-0)
lib/charmhelpers/core/services/__init__.py (+16/-0)
lib/charmhelpers/core/services/base.py (+59/-19)
lib/charmhelpers/core/services/helpers.py (+59/-10)
lib/charmhelpers/core/strutils.py (+72/-0)
lib/charmhelpers/core/sysctl.py (+28/-6)
lib/charmhelpers/core/templating.py (+37/-8)
lib/charmhelpers/core/unitdata.py (+521/-0)
lib/charmhelpers/fetch/__init__.py (+75/-22)
lib/charmhelpers/fetch/archiveurl.py (+34/-12)
lib/charmhelpers/fetch/bzrurl.py (+51/-28)
lib/charmhelpers/fetch/giturl.py (+45/-23)
lib/charmhelpers/payload/__init__.py (+16/-0)
lib/charmhelpers/payload/archive.py (+16/-0)
lib/charmhelpers/payload/execd.py (+16/-0)
metadata.yaml (+1/-0)
revision (+1/-1)
scripts/charm_helpers_sync.py (+38/-8)
tests/10-deploy-and-upgrade (+2/-22)
To merge this branch: bzr merge lp:~petevg/charms/trusty/mariadb/fixes-for-xenial
Reviewer Review Type Date Requested Status
Daniel Bartholomew 2016-07-12 Pending
Review via email: mp+299826@code.launchpad.net

Description of the change

Hi Daniel,

As part of a project to ship IBM a charm that works on their z Linux distro, I made a version of the mariadb charm that works on xenial.

The changes are:

1) Ported config-changed and install to python3 (python2 is not installed by default on xenial, though it does get installed when we install python-mysql in the config-changed script, so install and config-changed are really the only two scripts that needed porting).
2) Told the charm to install mariadb packages from universe on xenial. (This has the side effect of fixing an issue installing on z Linux -- if you'd like to avoid using the universe packages, I can change the check to install from universe only if we're on z Linux.)

To post a comment you must log in.
Pete Vander Giessen (petevg) wrote :

Just a quick note on this: part of what I did was sync up charm_helpers, to incorporate recent fixes that handle the case where Pyaml is not installed on the version of Python that you're using to execute a script (this makes the charm work on python3 in trusty).

That makes the PR look a lot bigger than it actually is.

Unmerged revisions

38. By Pete Vander Giessen <email address hidden> on 2016-07-12

Updated to be multi-series charm (trusty and xenial)

37. By Pete Vander Giessen <email address hidden> on 2016-07-08

Updated charm to work on xenial.

Python2 is not installed by default on xenial. Ported install and config-changed to python3 so that they will execute on a fresh xenial box. (The config-changed hook install python-mysql, which depends on python2, so the other hooks did not need to be ported).

Updated charmhelpers to latest version, so that missing dependencies, like Pyaml, will be installed (fixes an issue in trusty, where Pyaml is not installed by default in python3).

Charm now uses mariadb packages from universe in xenial, as those packages are up-to-date.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2016-06-03 17:18:00 +0000
3+++ README.md 2016-07-12 15:38:59 +0000
4@@ -11,78 +11,67 @@
5 MariaDB and enhances it with an optimized configuration, additional testing,
6 and available 24/7 professional support and consulting.
7
8-This charm deploys either MariaDB, using packages provided by the MariaDB
9-Foundation, including packages for IBM's POWER8 platform; or MariaDB
10-Enterprise, using packages in a repository provided by MariaDB Corporation,
11-Inc.
12+This charm deploys MariaDB using packages in a repository provided by the
13+MariaDB Foundation or, optionally, MariaDB Enterprise packages in a repository
14+provided by MariaDB Corporation, Inc.
15
16-Note: Packages for IBM's POWER8 platform are only available from the MariaDB
17-Foundation repository.
18+**Packages for IBM's Power8 platform are only available from the MariaDB
19+Enterprise repository.**
20
21 As much as possible this charm uses the same charm structure as the MySQL charm
22 for the sake of compatability.
23
24+
25 # Usage
26
27 ## General Usage
28
29-### Deploying MariaDB
30-To deploy MariaDB from the MariaDB Foundation, simply deploy like so:
31+To deploy a MariaDB service:
32
33 juju deploy mariadb
34
35-MariaDB will be deployed.
36-
37-### Deploying MariaDB Enterprise
38-To deploy a MariaDB Enterprise service, first login to the
39-[MariaDB Portal](https://mariadb.com/my_portal). On that page you will find
40-your MariaDB Enterprise download token. It is of the form `xxxx-xxxx`.
41+Once deployed, you can retrive the MariaDB root user password by logging in to
42+the machine via `juju ssh` and reading the `/var/lib/mysql/mysql.passwd` file.
43+To log in as root MariaDB User at the MariaDB console you can issue the
44+following:
45+
46+ juju ssh mariadb/0
47+ mysql -u root -p$(sudo cat /var/lib/mysql/mysql.passwd)
48+
49+## To deploy MariaDB Enterprise instead of MariaDB
50+
51+First obtain a username/password from the [MariaDB Portal](http://mariadb.com)
52+and you will then have access to the MariaDB Enterprise repository.
53
54 Next create a file called enterprise.yaml with the following contents,
55-replacing the placeholder token with your actual token:
56+replacing `username:password` with your actual username and password:
57
58 mariadb:
59 enterprise-eula: true
60- token: "xxxx-xxxx"
61+ key: 0xd324876ebe6a595f
62+ source: "deb https://username:password@code.mariadb.com/mariadb-enterprise/10.0/repo/ubuntu trusty main"
63
64-Lastly, deploy MariaDB Enterprise like so:
65+Lastly, deploy MariaDB as normal but with the addition of:
66
67 juju deploy --config ./enterprise.yaml mariadb
68
69-MariaDB Enterprise will be deployed. You must agree to all terms contained in
70+MariaDB Enterprise will be deployed instead of MariaDB. You must agree to all
71+terms contained in `ENTERPRISE-LICENSE.md` in the charm directory to use
72+MariaDB Enterprise.
73+
74+## To switch from MariaDB to MariaDB Enterprise
75+
76+If you deployed MariaDB and would like to switch to MariaDB Enterprise, first
77+obtain a username/password from the [MariaDB Portal](http://mariadb.com) and
78+you will then have access to the MariaDB Enterprise repository. You can then
79+enable the repository in the charm with the following configuration:
80+
81+ juju set mariadb enterprise-eula=true key="0xd324876ebe6a595f" source="deb https://username:password@code.mariadb.com/mariadb-enterprise/10.0/repo/ubuntu trusty main"
82+
83+This will perform an in-place binary upgrade on all the MariaDB nodes from
84+MariaDB to MariaDB Enterprise. You must agree to all terms contained in
85 `ENTERPRISE-LICENSE.md` in the charm directory to use MariaDB Enterprise.
86
87-### Installing a different series of MariaDB
88-Different series of MariaDB can be installed using the charm. The default
89-series is MariaDB 10.1, but the older MariaDB 5.5 and 10.0 are also available.
90-To install one of these older series, create a mariadb.yaml file with the
91-following contents (example is for MariaDB 5.5):
92-
93- mariadb:
94- series: "5.5"
95-
96-You could also add the series line to your enterprise.yaml file, if you are
97-using MariaDB Enterprise. Then when deploying MariaDB, reference the file like
98-so:
99-
100- juju deploy --config ./mariadb.yaml mariadb
101-
102-Warning: Using the set command to downgrade MariaDB after initial deployment, for example, like so:
103-
104- juju set mariadb series="5.5"
105-
106-...does not work. Using the set command to upgrade MariaDB; from 5.5 to 10.0,
107-or from 10.0 to 10.1; does work.
108-
109-
110-### After deploying
111-Once deployed, you can retrive the MariaDB root user password by logging in to
112-the machine via `juju ssh` and reading the `/var/lib/mysql/mysql.passwd` file.
113-To log in as the root MariaDB user at the MariaDB console you can, for example,
114-issue the following:
115-
116- juju ssh mariadb/0
117- mysql -u root -p$(sudo cat /var/lib/mysql/mysql.passwd)
118
119 # Scale Out Usage
120
121@@ -96,7 +85,7 @@
122 To deploy a slave:
123
124 # deploy second service
125- juju deploy --config ./enterprise.yaml mariadb mariadb-slave
126+ juju deploy mariadb mariadb-slave
127
128 # add master to slave relation
129 juju add-relation mariadb:master mariadb-slave:slave
130
131=== modified file 'config.yaml'
132--- config.yaml 2016-06-03 16:06:28 +0000
133+++ config.yaml 2016-07-12 15:38:59 +0000
134@@ -1,6 +1,6 @@
135 options:
136 dataset-size:
137- default: '50%'
138+ default: '80%'
139 description: |
140 How much data do you want to keep in memory in the database. This
141 will be used to tune settings in the database server appropriately.
142@@ -107,38 +107,13 @@
143 I have read and agree to the ENTERPRISE TRIAL agreement, located
144 in ENTERPRISE-LICENSE.md located in the charm, or on the web here:
145 https://mariadb.com/about/legal/evaluation-agreement
146- base-url:
147- type: string
148- default: "https://downloads.mariadb.com/enterprise"
149- description: |
150- Base URL of the MariaDB Enterprise repository package
151- base-url-org:
152- type: string
153- default: "http://ftp.osuosl.org/pub/mariadb/repo"
154- description: |
155- Base URL of the MariaDB repository
156- series:
157- type: string
158- default: "10.1"
159- description: |
160- Name of the MariaDB series to install
161- token:
162- type: string
163- default: ""
164- description: |
165- Enterprise download token from https://mariadb.com/my_portal
166- repo-pkg:
167- type: string
168- default: "mariadb-enterprise-repository.deb"
169- description: |
170- The name of the MariaDB Enterprise repository package
171+ source:
172+ type: string
173+ default: "deb http://ftp.osuosl.org/pub/mariadb/repo/10.0/ubuntu trusty main"
174+ description: |
175+ Repository Mirror string to install MariaDB from
176 key:
177- type: string
178- default: "0xce1a3dd5e3c94f49"
179- description: |
180- GPG Key used to verify MariaDB Enterprise packages
181- key-org:
182- type: string
183- default: "0xcbcb082a1bb943db 0xF1656F24C74CD1D8"
184- description: |
185- GPG Keys used to verify MariaDB packages
186+ type: string
187+ default: "0xcbcb082a1bb943db"
188+ description: |
189+ GPG Key used to verify apt packages.
190
191=== modified file 'hooks/common.py'
192--- hooks/common.py 2014-09-25 20:40:27 +0000
193+++ hooks/common.py 2016-07-12 15:38:59 +0000
194@@ -76,7 +76,7 @@
195 def create_database(db_name):
196 cursor = get_db_cursor()
197 try:
198- cursor.execute("CREATE DATABASE {}".format(db_name))
199+ cursor.execute("CREATE DATABASE `{}`".format(db_name))
200 finally:
201 cursor.close()
202
203@@ -99,7 +99,7 @@
204 remote_ip, password):
205 cursor = get_db_cursor()
206 try:
207- cursor.execute("GRANT ALL PRIVILEGES ON {}.* TO '{}'@'{}' "\
208+ cursor.execute("GRANT ALL PRIVILEGES ON `{}`.* TO '{}'@'{}' "\
209 "IDENTIFIED BY '{}'".format(db_name,
210 db_user,
211 remote_ip,
212
213=== modified file 'hooks/config-changed'
214--- hooks/config-changed 2016-05-31 20:31:40 +0000
215+++ hooks/config-changed 2016-07-12 15:38:59 +0000
216@@ -1,4 +1,4 @@
217-#!/usr/bin/python
218+#!/usr/bin/env python3
219
220 from subprocess import check_output, check_call, CalledProcessError
221 import tempfile
222@@ -8,17 +8,12 @@
223 import os
224 import sys
225 import platform
226-from string import upper
227 from subprocess import Popen, PIPE
228
229 sys.path.insert(0, os.path.join(os.environ['CHARM_DIR'], 'lib'))
230
231 from charmhelpers import fetch
232
233-from charmhelpers.core.host import (
234- lsb_release
235-)
236-
237 from charmhelpers.core import (
238 hookenv,
239 host,
240@@ -67,7 +62,8 @@
241 if IS_32BIT_SYSTEM:
242 log("32bit system restrictions in play", "INFO")
243
244-configs = json.loads(check_output(['config-get','--format=json']))
245+configs = json.loads(
246+ check_output(['config-get','--format=json']).decode('utf-8'))
247
248 def get_memtotal():
249 with open('/proc/meminfo') as meminfo_file:
250@@ -75,19 +71,13 @@
251 (key, mem) = line.split(':', 2)
252 if key == 'MemTotal':
253 (mtot, modifier) = mem.strip().split(' ')
254- return '%s%s' % (mtot, upper(modifier[0]))
255-
256-
257-
258-#source = config['source']
259+ return '%s%s' % (mtot, modifier[0].upper())
260+
261+
262+
263+source = config['source']
264 accepted = config['enterprise-eula']
265-key = config['key']
266-key_org = config['key-org']
267-base = config['base-url']
268-base_org = config['base-url-org']
269-token = config['token']
270-pkg = config['repo-pkg']
271-series = config['series']
272+_, _, series = platform.dist()
273
274
275 # set MariaDB Root Password
276@@ -97,31 +87,20 @@
277
278 # preseed debconf with our admin password
279 dconf = Popen(['debconf-set-selections'], stdin=PIPE)
280-dconf.stdin.write("%s %s/root_password password %s\n" % (package, package, root_pass))
281-dconf.stdin.write("%s %s/root_password_again password %s\n" % (package, package, root_pass))
282+dconf.stdin.write(("%s %s/root_password password %s\n" % (package, package, root_pass)).encode())
283+dconf.stdin.write(("%s %s/root_password_again password %s\n" % (package, package, root_pass)).encode())
284 dconf.communicate()
285 dconf.wait()
286
287 # assumption of mariadb packages being delivered from code.mariadb
288-if not accepted:
289- log('Enterprise EULA not accepted - installing from mariadb.org', 'INFO')
290- check_call(['apt-key', 'adv', '--keyserver',
291- 'hkp://keyserver.ubuntu.com:80',
292- '--recv', key_org])
293- release = lsb_release()['DISTRIB_CODENAME']
294- fetch.add_source("deb %s/%s/ubuntu %s main" % (base_org,series,release), None)
295- fetch.apt_update()
296- packages = ['mariadb-server', 'mariadb-client']
297- fetch.apt_install(packages)
298-
299+if not accepted and "code.mariadb" in source:
300+ log('EULA not accepted - doing nothing', 'WARNING')
301+ host.service_stop('mysql')
302 else:
303- check_call(['apt-key', 'adv', '--keyserver',
304- 'hkp://keyserver.ubuntu.com:80',
305- '--recv', key])
306- check_call(['wget', '-N',
307- "%s/%s/generate/%s/%s" % (base, token, series, pkg)])
308- check_call(['dpkg', '-i', 'mariadb-enterprise-repository.deb'])
309- fetch.apt_update()
310+ if series == 'trusty':
311+ # an up-to-date mariadb is in universe on xenial and newer.
312+ fetch.add_source(source, config['key'])
313+ fetch.apt_update()
314
315 packages = ['mariadb-server', 'mariadb-client']
316 fetch.apt_install(packages)
317@@ -234,7 +213,7 @@
318 #
319 # * Fine Tuning
320 #
321-key_buffer = %(key-buffer)s
322+key_buffer = %(key-buffer)s
323 max_allowed_packet = 16M
324 thread_stack = 192K
325 thread_cache_size = 8
326@@ -258,9 +237,9 @@
327 # As of 5.1 you can enable the log at runtime!
328 #general_log_file = /usr/local/mysql/data/mysql.log
329 #general_log = 1
330-#
331+#
332 # Error log - should be very few entries.
333-#
334+#
335 log_error = /var/log/mysql/error.log
336 #
337 # Here you can see queries with especially long duration
338@@ -310,7 +289,7 @@
339 #no-auto-rehash # faster start of mysql but no tab completition
340
341 [isamchk]
342-key_buffer = 16M
343+key_buffer = 16M
344
345 #
346 # * IMPORTANT: Additional settings that can override those from this file!
347@@ -343,7 +322,7 @@
348 }
349
350 need_restart = False
351-for target, content in targets.iteritems():
352+for target, content in targets.items():
353 tdir = os.path.dirname(target)
354 if len(content) == 0 and os.path.exists(target):
355 os.unlink(target)
356@@ -353,11 +332,11 @@
357 t.write(content)
358 t.flush()
359 tmd5 = hashlib.md5()
360- tmd5.update(content)
361+ tmd5.update(content.encode())
362 if os.path.exists(target):
363 with open(target, 'r') as old:
364 md5 = hashlib.md5()
365- md5.update(old.read())
366+ md5.update(old.read().encode())
367 oldhash = md5.digest()
368 if oldhash != tmd5.digest():
369 os.rename(target, '%s.%s' % (target, md5.hexdigest()))
370
371=== modified file 'hooks/install'
372--- hooks/install 2015-01-30 23:46:18 +0000
373+++ hooks/install 2016-07-12 15:38:59 +0000
374@@ -1,4 +1,4 @@
375-#!/usr/bin/env python
376+#!/usr/bin/env python3
377
378 import os
379 import sys
380@@ -37,7 +37,7 @@
381 os.makedirs(varpath)
382 with open(PASSFILE, 'a'):
383 os.utime(PASSFILE, None)
384- os.chmod(PASSFILE, 0600)
385+ os.chmod(PASSFILE, 0o600)
386 except:
387 pass
388 # Touch the passfile
389
390=== modified file 'hooks/start'
391--- hooks/start 2014-09-24 21:03:47 +0000
392+++ hooks/start 2016-07-12 15:38:59 +0000
393@@ -1,4 +1,5 @@
394 #!/bin/bash
395 set -e
396-/etc/init.d/mysql restart || /etc/init.d/mysql start
397+service mysql restart || service mysql start
398+
399
400
401=== modified file 'lib/charmhelpers/__init__.py'
402--- lib/charmhelpers/__init__.py 2014-12-02 19:35:26 +0000
403+++ lib/charmhelpers/__init__.py 2016-07-12 15:38:59 +0000
404@@ -0,0 +1,38 @@
405+# Copyright 2014-2015 Canonical Limited.
406+#
407+# This file is part of charm-helpers.
408+#
409+# charm-helpers is free software: you can redistribute it and/or modify
410+# it under the terms of the GNU Lesser General Public License version 3 as
411+# published by the Free Software Foundation.
412+#
413+# charm-helpers is distributed in the hope that it will be useful,
414+# but WITHOUT ANY WARRANTY; without even the implied warranty of
415+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
416+# GNU Lesser General Public License for more details.
417+#
418+# You should have received a copy of the GNU Lesser General Public License
419+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
420+
421+# Bootstrap charm-helpers, installing its dependencies if necessary using
422+# only standard libraries.
423+import subprocess
424+import sys
425+
426+try:
427+ import six # flake8: noqa
428+except ImportError:
429+ if sys.version_info.major == 2:
430+ subprocess.check_call(['apt-get', 'install', '-y', 'python-six'])
431+ else:
432+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-six'])
433+ import six # flake8: noqa
434+
435+try:
436+ import yaml # flake8: noqa
437+except ImportError:
438+ if sys.version_info.major == 2:
439+ subprocess.check_call(['apt-get', 'install', '-y', 'python-yaml'])
440+ else:
441+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
442+ import yaml # flake8: noqa
443
444=== modified file 'lib/charmhelpers/core/__init__.py'
445--- lib/charmhelpers/core/__init__.py 2014-12-02 19:35:26 +0000
446+++ lib/charmhelpers/core/__init__.py 2016-07-12 15:38:59 +0000
447@@ -0,0 +1,15 @@
448+# Copyright 2014-2015 Canonical Limited.
449+#
450+# This file is part of charm-helpers.
451+#
452+# charm-helpers is free software: you can redistribute it and/or modify
453+# it under the terms of the GNU Lesser General Public License version 3 as
454+# published by the Free Software Foundation.
455+#
456+# charm-helpers is distributed in the hope that it will be useful,
457+# but WITHOUT ANY WARRANTY; without even the implied warranty of
458+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
459+# GNU Lesser General Public License for more details.
460+#
461+# You should have received a copy of the GNU Lesser General Public License
462+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
463
464=== added file 'lib/charmhelpers/core/decorators.py'
465--- lib/charmhelpers/core/decorators.py 1970-01-01 00:00:00 +0000
466+++ lib/charmhelpers/core/decorators.py 2016-07-12 15:38:59 +0000
467@@ -0,0 +1,57 @@
468+# Copyright 2014-2015 Canonical Limited.
469+#
470+# This file is part of charm-helpers.
471+#
472+# charm-helpers is free software: you can redistribute it and/or modify
473+# it under the terms of the GNU Lesser General Public License version 3 as
474+# published by the Free Software Foundation.
475+#
476+# charm-helpers is distributed in the hope that it will be useful,
477+# but WITHOUT ANY WARRANTY; without even the implied warranty of
478+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
479+# GNU Lesser General Public License for more details.
480+#
481+# You should have received a copy of the GNU Lesser General Public License
482+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
483+
484+#
485+# Copyright 2014 Canonical Ltd.
486+#
487+# Authors:
488+# Edward Hope-Morley <opentastic@gmail.com>
489+#
490+
491+import time
492+
493+from charmhelpers.core.hookenv import (
494+ log,
495+ INFO,
496+)
497+
498+
499+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
500+ """If the decorated function raises exception exc_type, allow num_retries
501+ retry attempts before raise the exception.
502+ """
503+ def _retry_on_exception_inner_1(f):
504+ def _retry_on_exception_inner_2(*args, **kwargs):
505+ retries = num_retries
506+ multiplier = 1
507+ while True:
508+ try:
509+ return f(*args, **kwargs)
510+ except exc_type:
511+ if not retries:
512+ raise
513+
514+ delay = base_delay * multiplier
515+ multiplier += 1
516+ log("Retrying '%s' %d more times (delay=%s)" %
517+ (f.__name__, retries, delay), level=INFO)
518+ retries -= 1
519+ if delay:
520+ time.sleep(delay)
521+
522+ return _retry_on_exception_inner_2
523+
524+ return _retry_on_exception_inner_1
525
526=== added file 'lib/charmhelpers/core/files.py'
527--- lib/charmhelpers/core/files.py 1970-01-01 00:00:00 +0000
528+++ lib/charmhelpers/core/files.py 2016-07-12 15:38:59 +0000
529@@ -0,0 +1,45 @@
530+#!/usr/bin/env python
531+# -*- coding: utf-8 -*-
532+
533+# Copyright 2014-2015 Canonical Limited.
534+#
535+# This file is part of charm-helpers.
536+#
537+# charm-helpers is free software: you can redistribute it and/or modify
538+# it under the terms of the GNU Lesser General Public License version 3 as
539+# published by the Free Software Foundation.
540+#
541+# charm-helpers is distributed in the hope that it will be useful,
542+# but WITHOUT ANY WARRANTY; without even the implied warranty of
543+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
544+# GNU Lesser General Public License for more details.
545+#
546+# You should have received a copy of the GNU Lesser General Public License
547+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
548+
549+__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'
550+
551+import os
552+import subprocess
553+
554+
555+def sed(filename, before, after, flags='g'):
556+ """
557+ Search and replaces the given pattern on filename.
558+
559+ :param filename: relative or absolute file path.
560+ :param before: expression to be replaced (see 'man sed')
561+ :param after: expression to replace with (see 'man sed')
562+ :param flags: sed-compatible regex flags in example, to make
563+ the search and replace case insensitive, specify ``flags="i"``.
564+ The ``g`` flag is always specified regardless, so you do not
565+ need to remember to include it when overriding this parameter.
566+ :returns: If the sed command exit code was zero then return,
567+ otherwise raise CalledProcessError.
568+ """
569+ expression = r's/{0}/{1}/{2}'.format(before,
570+ after, flags)
571+
572+ return subprocess.check_call(["sed", "-i", "-r", "-e",
573+ expression,
574+ os.path.expanduser(filename)])
575
576=== modified file 'lib/charmhelpers/core/fstab.py'
577--- lib/charmhelpers/core/fstab.py 2014-12-02 19:35:26 +0000
578+++ lib/charmhelpers/core/fstab.py 2016-07-12 15:38:59 +0000
579@@ -1,11 +1,27 @@
580 #!/usr/bin/env python
581 # -*- coding: utf-8 -*-
582
583-__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
584+# Copyright 2014-2015 Canonical Limited.
585+#
586+# This file is part of charm-helpers.
587+#
588+# charm-helpers is free software: you can redistribute it and/or modify
589+# it under the terms of the GNU Lesser General Public License version 3 as
590+# published by the Free Software Foundation.
591+#
592+# charm-helpers is distributed in the hope that it will be useful,
593+# but WITHOUT ANY WARRANTY; without even the implied warranty of
594+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
595+# GNU Lesser General Public License for more details.
596+#
597+# You should have received a copy of the GNU Lesser General Public License
598+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
599
600 import io
601 import os
602
603+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
604+
605
606 class Fstab(io.FileIO):
607 """This class extends file in order to implement a file reader/writer
608@@ -61,7 +77,7 @@
609 for line in self.readlines():
610 line = line.decode('us-ascii')
611 try:
612- if line.strip() and not line.startswith("#"):
613+ if line.strip() and not line.strip().startswith("#"):
614 yield self._hydrate_entry(line)
615 except ValueError:
616 pass
617@@ -88,7 +104,7 @@
618
619 found = False
620 for index, line in enumerate(lines):
621- if not line.startswith("#"):
622+ if line.strip() and not line.strip().startswith("#"):
623 if self._hydrate_entry(line) == entry:
624 found = True
625 break
626
627=== modified file 'lib/charmhelpers/core/hookenv.py'
628--- lib/charmhelpers/core/hookenv.py 2014-12-02 19:35:26 +0000
629+++ lib/charmhelpers/core/hookenv.py 2016-07-12 15:38:59 +0000
630@@ -1,14 +1,37 @@
631+# Copyright 2014-2015 Canonical Limited.
632+#
633+# This file is part of charm-helpers.
634+#
635+# charm-helpers is free software: you can redistribute it and/or modify
636+# it under the terms of the GNU Lesser General Public License version 3 as
637+# published by the Free Software Foundation.
638+#
639+# charm-helpers is distributed in the hope that it will be useful,
640+# but WITHOUT ANY WARRANTY; without even the implied warranty of
641+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
642+# GNU Lesser General Public License for more details.
643+#
644+# You should have received a copy of the GNU Lesser General Public License
645+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
646+
647 "Interactions with the Juju environment"
648 # Copyright 2013 Canonical Ltd.
649 #
650 # Authors:
651 # Charm Helpers Developers <juju@lists.ubuntu.com>
652
653+from __future__ import print_function
654+import copy
655+from distutils.version import LooseVersion
656+from functools import wraps
657+import glob
658 import os
659 import json
660 import yaml
661 import subprocess
662 import sys
663+import errno
664+import tempfile
665 from subprocess import CalledProcessError
666
667 import six
668@@ -40,15 +63,18 @@
669
670 will cache the result of unit_get + 'test' for future calls.
671 """
672+ @wraps(func)
673 def wrapper(*args, **kwargs):
674 global cache
675 key = str((func, args, kwargs))
676 try:
677 return cache[key]
678 except KeyError:
679- res = func(*args, **kwargs)
680- cache[key] = res
681- return res
682+ pass # Drop out of the exception handler scope.
683+ res = func(*args, **kwargs)
684+ cache[key] = res
685+ return res
686+ wrapper._wrapped = func
687 return wrapper
688
689
690@@ -68,8 +94,21 @@
691 command = ['juju-log']
692 if level:
693 command += ['-l', level]
694+ if not isinstance(message, six.string_types):
695+ message = repr(message)
696 command += [message]
697- subprocess.call(command)
698+ # Missing juju-log should not cause failures in unit tests
699+ # Send log output to stderr
700+ try:
701+ subprocess.call(command)
702+ except OSError as e:
703+ if e.errno == errno.ENOENT:
704+ if level:
705+ message = "{}: {}".format(level, message)
706+ message = "juju-log: {}".format(message)
707+ print(message, file=sys.stderr)
708+ else:
709+ raise
710
711
712 class Serializable(UserDict):
713@@ -135,9 +174,19 @@
714 return os.environ.get('JUJU_RELATION', None)
715
716
717-def relation_id():
718- """The relation ID for the current relation hook"""
719- return os.environ.get('JUJU_RELATION_ID', None)
720+@cached
721+def relation_id(relation_name=None, service_or_unit=None):
722+ """The relation ID for the current or a specified relation"""
723+ if not relation_name and not service_or_unit:
724+ return os.environ.get('JUJU_RELATION_ID', None)
725+ elif relation_name and service_or_unit:
726+ service_name = service_or_unit.split('/')[0]
727+ for relid in relation_ids(relation_name):
728+ remote_service = remote_service_name(relid)
729+ if remote_service == service_name:
730+ return relid
731+ else:
732+ raise ValueError('Must specify neither or both of relation_name and service_or_unit')
733
734
735 def local_unit():
736@@ -147,7 +196,7 @@
737
738 def remote_unit():
739 """The remote unit for the current relation hook"""
740- return os.environ['JUJU_REMOTE_UNIT']
741+ return os.environ.get('JUJU_REMOTE_UNIT', None)
742
743
744 def service_name():
745@@ -155,9 +204,20 @@
746 return local_unit().split('/')[0]
747
748
749+@cached
750+def remote_service_name(relid=None):
751+ """The remote service name for a given relation-id (or the current relation)"""
752+ if relid is None:
753+ unit = remote_unit()
754+ else:
755+ units = related_units(relid)
756+ unit = units[0] if units else None
757+ return unit.split('/')[0] if unit else None
758+
759+
760 def hook_name():
761 """The name of the currently executing hook"""
762- return os.path.basename(sys.argv[0])
763+ return os.environ.get('JUJU_HOOK_NAME', os.path.basename(sys.argv[0]))
764
765
766 class Config(dict):
767@@ -207,23 +267,7 @@
768 self.path = os.path.join(charm_dir(), Config.CONFIG_FILE_NAME)
769 if os.path.exists(self.path):
770 self.load_previous()
771-
772- def __getitem__(self, key):
773- """For regular dict lookups, check the current juju config first,
774- then the previous (saved) copy. This ensures that user-saved values
775- will be returned by a dict lookup.
776-
777- """
778- try:
779- return dict.__getitem__(self, key)
780- except KeyError:
781- return (self._prev_dict or {})[key]
782-
783- def keys(self):
784- prev_keys = []
785- if self._prev_dict is not None:
786- prev_keys = self._prev_dict.keys()
787- return list(set(prev_keys + list(dict.keys(self))))
788+ atexit(self._implicit_save)
789
790 def load_previous(self, path=None):
791 """Load previous copy of config from disk.
792@@ -242,6 +286,9 @@
793 self.path = path or self.path
794 with open(self.path) as f:
795 self._prev_dict = json.load(f)
796+ for k, v in copy.deepcopy(self._prev_dict).items():
797+ if k not in self:
798+ self[k] = v
799
800 def changed(self, key):
801 """Return True if the current value for this key is different from
802@@ -273,13 +320,13 @@
803 instance.
804
805 """
806- if self._prev_dict:
807- for k, v in six.iteritems(self._prev_dict):
808- if k not in self:
809- self[k] = v
810 with open(self.path, 'w') as f:
811 json.dump(self, f)
812
813+ def _implicit_save(self):
814+ if self.implicit_save:
815+ self.save()
816+
817
818 @cached
819 def config(scope=None):
820@@ -322,18 +369,49 @@
821 """Set relation information for the current unit"""
822 relation_settings = relation_settings if relation_settings else {}
823 relation_cmd_line = ['relation-set']
824+ accepts_file = "--file" in subprocess.check_output(
825+ relation_cmd_line + ["--help"], universal_newlines=True)
826 if relation_id is not None:
827 relation_cmd_line.extend(('-r', relation_id))
828- for k, v in (list(relation_settings.items()) + list(kwargs.items())):
829- if v is None:
830- relation_cmd_line.append('{}='.format(k))
831- else:
832- relation_cmd_line.append('{}={}'.format(k, v))
833- subprocess.check_call(relation_cmd_line)
834+ settings = relation_settings.copy()
835+ settings.update(kwargs)
836+ for key, value in settings.items():
837+ # Force value to be a string: it always should, but some call
838+ # sites pass in things like dicts or numbers.
839+ if value is not None:
840+ settings[key] = "{}".format(value)
841+ if accepts_file:
842+ # --file was introduced in Juju 1.23.2. Use it by default if
843+ # available, since otherwise we'll break if the relation data is
844+ # too big. Ideally we should tell relation-set to read the data from
845+ # stdin, but that feature is broken in 1.23.2: Bug #1454678.
846+ with tempfile.NamedTemporaryFile(delete=False) as settings_file:
847+ settings_file.write(yaml.safe_dump(settings).encode("utf-8"))
848+ subprocess.check_call(
849+ relation_cmd_line + ["--file", settings_file.name])
850+ os.remove(settings_file.name)
851+ else:
852+ for key, value in settings.items():
853+ if value is None:
854+ relation_cmd_line.append('{}='.format(key))
855+ else:
856+ relation_cmd_line.append('{}={}'.format(key, value))
857+ subprocess.check_call(relation_cmd_line)
858 # Flush cache of any relation-gets for local unit
859 flush(local_unit())
860
861
862+def relation_clear(r_id=None):
863+ ''' Clears any relation data already set on relation r_id '''
864+ settings = relation_get(rid=r_id,
865+ unit=local_unit())
866+ for setting in settings:
867+ if setting not in ['public-address', 'private-address']:
868+ settings[setting] = None
869+ relation_set(relation_id=r_id,
870+ **settings)
871+
872+
873 @cached
874 def relation_ids(reltype=None):
875 """A list of relation_ids"""
876@@ -394,21 +472,101 @@
877
878
879 @cached
880+def metadata():
881+ """Get the current charm metadata.yaml contents as a python object"""
882+ with open(os.path.join(charm_dir(), 'metadata.yaml')) as md:
883+ return yaml.safe_load(md)
884+
885+
886+@cached
887 def relation_types():
888 """Get a list of relation types supported by this charm"""
889- charmdir = os.environ.get('CHARM_DIR', '')
890- mdf = open(os.path.join(charmdir, 'metadata.yaml'))
891- md = yaml.safe_load(mdf)
892 rel_types = []
893+ md = metadata()
894 for key in ('provides', 'requires', 'peers'):
895 section = md.get(key)
896 if section:
897 rel_types.extend(section.keys())
898- mdf.close()
899 return rel_types
900
901
902 @cached
903+def peer_relation_id():
904+ '''Get the peers relation id if a peers relation has been joined, else None.'''
905+ md = metadata()
906+ section = md.get('peers')
907+ if section:
908+ for key in section:
909+ relids = relation_ids(key)
910+ if relids:
911+ return relids[0]
912+ return None
913+
914+
915+@cached
916+def relation_to_interface(relation_name):
917+ """
918+ Given the name of a relation, return the interface that relation uses.
919+
920+ :returns: The interface name, or ``None``.
921+ """
922+ return relation_to_role_and_interface(relation_name)[1]
923+
924+
925+@cached
926+def relation_to_role_and_interface(relation_name):
927+ """
928+ Given the name of a relation, return the role and the name of the interface
929+ that relation uses (where role is one of ``provides``, ``requires``, or ``peers``).
930+
931+ :returns: A tuple containing ``(role, interface)``, or ``(None, None)``.
932+ """
933+ _metadata = metadata()
934+ for role in ('provides', 'requires', 'peers'):
935+ interface = _metadata.get(role, {}).get(relation_name, {}).get('interface')
936+ if interface:
937+ return role, interface
938+ return None, None
939+
940+
941+@cached
942+def role_and_interface_to_relations(role, interface_name):
943+ """
944+ Given a role and interface name, return a list of relation names for the
945+ current charm that use that interface under that role (where role is one
946+ of ``provides``, ``requires``, or ``peers``).
947+
948+ :returns: A list of relation names.
949+ """
950+ _metadata = metadata()
951+ results = []
952+ for relation_name, relation in _metadata.get(role, {}).items():
953+ if relation['interface'] == interface_name:
954+ results.append(relation_name)
955+ return results
956+
957+
958+@cached
959+def interface_to_relations(interface_name):
960+ """
961+ Given an interface, return a list of relation names for the current
962+ charm that use that interface.
963+
964+ :returns: A list of relation names.
965+ """
966+ results = []
967+ for role in ('provides', 'requires', 'peers'):
968+ results.extend(role_and_interface_to_relations(role, interface_name))
969+ return results
970+
971+
972+@cached
973+def charm_name():
974+ """Get the name of the current charm as is specified on metadata.yaml"""
975+ return metadata().get('name')
976+
977+
978+@cached
979 def relations():
980 """Get a nested dictionary of relation data for all related units"""
981 rels = {}
982@@ -468,11 +626,48 @@
983 return None
984
985
986+def unit_public_ip():
987+ """Get this unit's public IP address"""
988+ return unit_get('public-address')
989+
990+
991 def unit_private_ip():
992 """Get this unit's private IP address"""
993 return unit_get('private-address')
994
995
996+@cached
997+def storage_get(attribute=None, storage_id=None):
998+ """Get storage attributes"""
999+ _args = ['storage-get', '--format=json']
1000+ if storage_id:
1001+ _args.extend(('-s', storage_id))
1002+ if attribute:
1003+ _args.append(attribute)
1004+ try:
1005+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1006+ except ValueError:
1007+ return None
1008+
1009+
1010+@cached
1011+def storage_list(storage_name=None):
1012+ """List the storage IDs for the unit"""
1013+ _args = ['storage-list', '--format=json']
1014+ if storage_name:
1015+ _args.append(storage_name)
1016+ try:
1017+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
1018+ except ValueError:
1019+ return None
1020+ except OSError as e:
1021+ import errno
1022+ if e.errno == errno.ENOENT:
1023+ # storage-list does not exist
1024+ return []
1025+ raise
1026+
1027+
1028 class UnregisteredHookError(Exception):
1029 """Raised when an undefined hook is called"""
1030 pass
1031@@ -500,10 +695,14 @@
1032 hooks.execute(sys.argv)
1033 """
1034
1035- def __init__(self, config_save=True):
1036+ def __init__(self, config_save=None):
1037 super(Hooks, self).__init__()
1038 self._hooks = {}
1039- self._config_save = config_save
1040+
1041+ # For unknown reasons, we allow the Hooks constructor to override
1042+ # config().implicit_save.
1043+ if config_save is not None:
1044+ config().implicit_save = config_save
1045
1046 def register(self, name, function):
1047 """Register a hook"""
1048@@ -511,13 +710,16 @@
1049
1050 def execute(self, args):
1051 """Execute a registered hook based on args[0]"""
1052+ _run_atstart()
1053 hook_name = os.path.basename(args[0])
1054 if hook_name in self._hooks:
1055- self._hooks[hook_name]()
1056- if self._config_save:
1057- cfg = config()
1058- if cfg.implicit_save:
1059- cfg.save()
1060+ try:
1061+ self._hooks[hook_name]()
1062+ except SystemExit as x:
1063+ if x.code is None or x.code == 0:
1064+ _run_atexit()
1065+ raise
1066+ _run_atexit()
1067 else:
1068 raise UnregisteredHookError(hook_name)
1069
1070@@ -538,3 +740,270 @@
1071 def charm_dir():
1072 """Return the root directory of the current charm"""
1073 return os.environ.get('CHARM_DIR')
1074+
1075+
1076+@cached
1077+def action_get(key=None):
1078+ """Gets the value of an action parameter, or all key/value param pairs"""
1079+ cmd = ['action-get']
1080+ if key is not None:
1081+ cmd.append(key)
1082+ cmd.append('--format=json')
1083+ action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1084+ return action_data
1085+
1086+
1087+def action_set(values):
1088+ """Sets the values to be returned after the action finishes"""
1089+ cmd = ['action-set']
1090+ for k, v in list(values.items()):
1091+ cmd.append('{}={}'.format(k, v))
1092+ subprocess.check_call(cmd)
1093+
1094+
1095+def action_fail(message):
1096+ """Sets the action status to failed and sets the error message.
1097+
1098+ The results set by action_set are preserved."""
1099+ subprocess.check_call(['action-fail', message])
1100+
1101+
1102+def action_name():
1103+ """Get the name of the currently executing action."""
1104+ return os.environ.get('JUJU_ACTION_NAME')
1105+
1106+
1107+def action_uuid():
1108+ """Get the UUID of the currently executing action."""
1109+ return os.environ.get('JUJU_ACTION_UUID')
1110+
1111+
1112+def action_tag():
1113+ """Get the tag for the currently executing action."""
1114+ return os.environ.get('JUJU_ACTION_TAG')
1115+
1116+
1117+def status_set(workload_state, message):
1118+ """Set the workload state with a message
1119+
1120+ Use status-set to set the workload state with a message which is visible
1121+ to the user via juju status. If the status-set command is not found then
1122+ assume this is juju < 1.23 and juju-log the message unstead.
1123+
1124+ workload_state -- valid juju workload state.
1125+ message -- status update message
1126+ """
1127+ valid_states = ['maintenance', 'blocked', 'waiting', 'active']
1128+ if workload_state not in valid_states:
1129+ raise ValueError(
1130+ '{!r} is not a valid workload state'.format(workload_state)
1131+ )
1132+ cmd = ['status-set', workload_state, message]
1133+ try:
1134+ ret = subprocess.call(cmd)
1135+ if ret == 0:
1136+ return
1137+ except OSError as e:
1138+ if e.errno != errno.ENOENT:
1139+ raise
1140+ log_message = 'status-set failed: {} {}'.format(workload_state,
1141+ message)
1142+ log(log_message, level='INFO')
1143+
1144+
1145+def status_get():
1146+ """Retrieve the previously set juju workload state and message
1147+
1148+ If the status-get command is not found then assume this is juju < 1.23 and
1149+ return 'unknown', ""
1150+
1151+ """
1152+ cmd = ['status-get', "--format=json", "--include-data"]
1153+ try:
1154+ raw_status = subprocess.check_output(cmd)
1155+ except OSError as e:
1156+ if e.errno == errno.ENOENT:
1157+ return ('unknown', "")
1158+ else:
1159+ raise
1160+ else:
1161+ status = json.loads(raw_status.decode("UTF-8"))
1162+ return (status["status"], status["message"])
1163+
1164+
1165+def translate_exc(from_exc, to_exc):
1166+ def inner_translate_exc1(f):
1167+ @wraps(f)
1168+ def inner_translate_exc2(*args, **kwargs):
1169+ try:
1170+ return f(*args, **kwargs)
1171+ except from_exc:
1172+ raise to_exc
1173+
1174+ return inner_translate_exc2
1175+
1176+ return inner_translate_exc1
1177+
1178+
1179+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1180+def is_leader():
1181+ """Does the current unit hold the juju leadership
1182+
1183+ Uses juju to determine whether the current unit is the leader of its peers
1184+ """
1185+ cmd = ['is-leader', '--format=json']
1186+ return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1187+
1188+
1189+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1190+def leader_get(attribute=None):
1191+ """Juju leader get value(s)"""
1192+ cmd = ['leader-get', '--format=json'] + [attribute or '-']
1193+ return json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1194+
1195+
1196+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1197+def leader_set(settings=None, **kwargs):
1198+ """Juju leader set value(s)"""
1199+ # Don't log secrets.
1200+ # log("Juju leader-set '%s'" % (settings), level=DEBUG)
1201+ cmd = ['leader-set']
1202+ settings = settings or {}
1203+ settings.update(kwargs)
1204+ for k, v in settings.items():
1205+ if v is None:
1206+ cmd.append('{}='.format(k))
1207+ else:
1208+ cmd.append('{}={}'.format(k, v))
1209+ subprocess.check_call(cmd)
1210+
1211+
1212+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1213+def payload_register(ptype, klass, pid):
1214+ """ is used while a hook is running to let Juju know that a
1215+ payload has been started."""
1216+ cmd = ['payload-register']
1217+ for x in [ptype, klass, pid]:
1218+ cmd.append(x)
1219+ subprocess.check_call(cmd)
1220+
1221+
1222+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1223+def payload_unregister(klass, pid):
1224+ """ is used while a hook is running to let Juju know
1225+ that a payload has been manually stopped. The <class> and <id> provided
1226+ must match a payload that has been previously registered with juju using
1227+ payload-register."""
1228+ cmd = ['payload-unregister']
1229+ for x in [klass, pid]:
1230+ cmd.append(x)
1231+ subprocess.check_call(cmd)
1232+
1233+
1234+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1235+def payload_status_set(klass, pid, status):
1236+ """is used to update the current status of a registered payload.
1237+ The <class> and <id> provided must match a payload that has been previously
1238+ registered with juju using payload-register. The <status> must be one of the
1239+ follow: starting, started, stopping, stopped"""
1240+ cmd = ['payload-status-set']
1241+ for x in [klass, pid, status]:
1242+ cmd.append(x)
1243+ subprocess.check_call(cmd)
1244+
1245+
1246+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1247+def resource_get(name):
1248+ """used to fetch the resource path of the given name.
1249+
1250+ <name> must match a name of defined resource in metadata.yaml
1251+
1252+ returns either a path or False if resource not available
1253+ """
1254+ if not name:
1255+ return False
1256+
1257+ cmd = ['resource-get', name]
1258+ try:
1259+ return subprocess.check_output(cmd).decode('UTF-8')
1260+ except subprocess.CalledProcessError:
1261+ return False
1262+
1263+
1264+@cached
1265+def juju_version():
1266+ """Full version string (eg. '1.23.3.1-trusty-amd64')"""
1267+ # Per https://bugs.launchpad.net/juju-core/+bug/1455368/comments/1
1268+ jujud = glob.glob('/var/lib/juju/tools/machine-*/jujud')[0]
1269+ return subprocess.check_output([jujud, 'version'],
1270+ universal_newlines=True).strip()
1271+
1272+
1273+@cached
1274+def has_juju_version(minimum_version):
1275+ """Return True if the Juju version is at least the provided version"""
1276+ return LooseVersion(juju_version()) >= LooseVersion(minimum_version)
1277+
1278+
1279+_atexit = []
1280+_atstart = []
1281+
1282+
1283+def atstart(callback, *args, **kwargs):
1284+ '''Schedule a callback to run before the main hook.
1285+
1286+ Callbacks are run in the order they were added.
1287+
1288+ This is useful for modules and classes to perform initialization
1289+ and inject behavior. In particular:
1290+
1291+ - Run common code before all of your hooks, such as logging
1292+ the hook name or interesting relation data.
1293+ - Defer object or module initialization that requires a hook
1294+ context until we know there actually is a hook context,
1295+ making testing easier.
1296+ - Rather than requiring charm authors to include boilerplate to
1297+ invoke your helper's behavior, have it run automatically if
1298+ your object is instantiated or module imported.
1299+
1300+ This is not at all useful after your hook framework as been launched.
1301+ '''
1302+ global _atstart
1303+ _atstart.append((callback, args, kwargs))
1304+
1305+
1306+def atexit(callback, *args, **kwargs):
1307+ '''Schedule a callback to run on successful hook completion.
1308+
1309+ Callbacks are run in the reverse order that they were added.'''
1310+ _atexit.append((callback, args, kwargs))
1311+
1312+
1313+def _run_atstart():
1314+ '''Hook frameworks must invoke this before running the main hook body.'''
1315+ global _atstart
1316+ for callback, args, kwargs in _atstart:
1317+ callback(*args, **kwargs)
1318+ del _atstart[:]
1319+
1320+
1321+def _run_atexit():
1322+ '''Hook frameworks must invoke this after the main hook body has
1323+ successfully completed. Do not invoke it if the hook fails.'''
1324+ global _atexit
1325+ for callback, args, kwargs in reversed(_atexit):
1326+ callback(*args, **kwargs)
1327+ del _atexit[:]
1328+
1329+
1330+@translate_exc(from_exc=OSError, to_exc=NotImplementedError)
1331+def network_get_primary_address(binding):
1332+ '''
1333+ Retrieve the primary network address for a named binding
1334+
1335+ :param binding: string. The name of a relation of extra-binding
1336+ :return: string. The primary IP address for the named binding
1337+ :raise: NotImplementedError if run on Juju < 2.0
1338+ '''
1339+ cmd = ['network-get', '--primary-address', binding]
1340+ return subprocess.check_output(cmd).decode('UTF-8').strip()
1341
1342=== modified file 'lib/charmhelpers/core/host.py'
1343--- lib/charmhelpers/core/host.py 2014-12-02 19:35:26 +0000
1344+++ lib/charmhelpers/core/host.py 2016-07-12 15:38:59 +0000
1345@@ -1,3 +1,19 @@
1346+# Copyright 2014-2015 Canonical Limited.
1347+#
1348+# This file is part of charm-helpers.
1349+#
1350+# charm-helpers is free software: you can redistribute it and/or modify
1351+# it under the terms of the GNU Lesser General Public License version 3 as
1352+# published by the Free Software Foundation.
1353+#
1354+# charm-helpers is distributed in the hope that it will be useful,
1355+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1356+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1357+# GNU Lesser General Public License for more details.
1358+#
1359+# You should have received a copy of the GNU Lesser General Public License
1360+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1361+
1362 """Tools for working with the host system"""
1363 # Copyright 2012 Canonical Ltd.
1364 #
1365@@ -8,11 +24,14 @@
1366 import os
1367 import re
1368 import pwd
1369+import glob
1370 import grp
1371 import random
1372 import string
1373 import subprocess
1374 import hashlib
1375+import functools
1376+import itertools
1377 from contextlib import contextmanager
1378 from collections import OrderedDict
1379
1380@@ -46,25 +65,94 @@
1381 return service_result
1382
1383
1384+def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"):
1385+ """Pause a system service.
1386+
1387+ Stop it, and prevent it from starting again at boot."""
1388+ stopped = True
1389+ if service_running(service_name):
1390+ stopped = service_stop(service_name)
1391+ upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
1392+ sysv_file = os.path.join(initd_dir, service_name)
1393+ if init_is_systemd():
1394+ service('disable', service_name)
1395+ elif os.path.exists(upstart_file):
1396+ override_path = os.path.join(
1397+ init_dir, '{}.override'.format(service_name))
1398+ with open(override_path, 'w') as fh:
1399+ fh.write("manual\n")
1400+ elif os.path.exists(sysv_file):
1401+ subprocess.check_call(["update-rc.d", service_name, "disable"])
1402+ else:
1403+ raise ValueError(
1404+ "Unable to detect {0} as SystemD, Upstart {1} or"
1405+ " SysV {2}".format(
1406+ service_name, upstart_file, sysv_file))
1407+ return stopped
1408+
1409+
1410+def service_resume(service_name, init_dir="/etc/init",
1411+ initd_dir="/etc/init.d"):
1412+ """Resume a system service.
1413+
1414+ Reenable starting again at boot. Start the service"""
1415+ upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
1416+ sysv_file = os.path.join(initd_dir, service_name)
1417+ if init_is_systemd():
1418+ service('enable', service_name)
1419+ elif os.path.exists(upstart_file):
1420+ override_path = os.path.join(
1421+ init_dir, '{}.override'.format(service_name))
1422+ if os.path.exists(override_path):
1423+ os.unlink(override_path)
1424+ elif os.path.exists(sysv_file):
1425+ subprocess.check_call(["update-rc.d", service_name, "enable"])
1426+ else:
1427+ raise ValueError(
1428+ "Unable to detect {0} as SystemD, Upstart {1} or"
1429+ " SysV {2}".format(
1430+ service_name, upstart_file, sysv_file))
1431+
1432+ started = service_running(service_name)
1433+ if not started:
1434+ started = service_start(service_name)
1435+ return started
1436+
1437+
1438 def service(action, service_name):
1439 """Control a system service"""
1440- cmd = ['service', service_name, action]
1441+ if init_is_systemd():
1442+ cmd = ['systemctl', action, service_name]
1443+ else:
1444+ cmd = ['service', service_name, action]
1445 return subprocess.call(cmd) == 0
1446
1447
1448-def service_running(service):
1449+_UPSTART_CONF = "/etc/init/{}.conf"
1450+_INIT_D_CONF = "/etc/init.d/{}"
1451+
1452+
1453+def service_running(service_name):
1454 """Determine whether a system service is running"""
1455- try:
1456- output = subprocess.check_output(
1457- ['service', service, 'status'],
1458- stderr=subprocess.STDOUT).decode('UTF-8')
1459- except subprocess.CalledProcessError:
1460+ if init_is_systemd():
1461+ return service('is-active', service_name)
1462+ else:
1463+ if os.path.exists(_UPSTART_CONF.format(service_name)):
1464+ try:
1465+ output = subprocess.check_output(
1466+ ['status', service_name],
1467+ stderr=subprocess.STDOUT).decode('UTF-8')
1468+ except subprocess.CalledProcessError:
1469+ return False
1470+ else:
1471+ # This works for upstart scripts where the 'service' command
1472+ # returns a consistent string to represent running 'start/running'
1473+ if "start/running" in output:
1474+ return True
1475+ elif os.path.exists(_INIT_D_CONF.format(service_name)):
1476+ # Check System V scripts init script return codes
1477+ return service('status', service_name)
1478 return False
1479- else:
1480- if ("start/running" in output or "is running" in output):
1481- return True
1482- else:
1483- return False
1484
1485
1486 def service_available(service_name):
1487@@ -74,19 +162,46 @@
1488 ['service', service_name, 'status'],
1489 stderr=subprocess.STDOUT).decode('UTF-8')
1490 except subprocess.CalledProcessError as e:
1491- return 'unrecognized service' not in e.output
1492+ return b'unrecognized service' not in e.output
1493 else:
1494 return True
1495
1496
1497-def adduser(username, password=None, shell='/bin/bash', system_user=False):
1498- """Add a user to the system"""
1499+SYSTEMD_SYSTEM = '/run/systemd/system'
1500+
1501+
1502+def init_is_systemd():
1503+ """Return True if the host system uses systemd, False otherwise."""
1504+ return os.path.isdir(SYSTEMD_SYSTEM)
1505+
1506+
1507+def adduser(username, password=None, shell='/bin/bash', system_user=False,
1508+ primary_group=None, secondary_groups=None, uid=None):
1509+ """Add a user to the system.
1510+
1511+ Will log but otherwise succeed if the user already exists.
1512+
1513+ :param str username: Username to create
1514+ :param str password: Password for user; if ``None``, create a system user
1515+ :param str shell: The default shell for the user
1516+ :param bool system_user: Whether to create a login or system user
1517+ :param str primary_group: Primary group for user; defaults to username
1518+ :param list secondary_groups: Optional list of additional groups
1519+ :param int uid: UID for user being created
1520+
1521+ :returns: The password database entry struct, as returned by `pwd.getpwnam`
1522+ """
1523 try:
1524 user_info = pwd.getpwnam(username)
1525 log('user {0} already exists!'.format(username))
1526+ if uid:
1527+ user_info = pwd.getpwuid(int(uid))
1528+ log('user with uid {0} already exists!'.format(uid))
1529 except KeyError:
1530 log('creating user {0}'.format(username))
1531 cmd = ['useradd']
1532+ if uid:
1533+ cmd.extend(['--uid', str(uid)])
1534 if system_user or password is None:
1535 cmd.append('--system')
1536 else:
1537@@ -95,19 +210,99 @@
1538 '--shell', shell,
1539 '--password', password,
1540 ])
1541+ if not primary_group:
1542+ try:
1543+ grp.getgrnam(username)
1544+ primary_group = username # avoid "group exists" error
1545+ except KeyError:
1546+ pass
1547+ if primary_group:
1548+ cmd.extend(['-g', primary_group])
1549+ if secondary_groups:
1550+ cmd.extend(['-G', ','.join(secondary_groups)])
1551 cmd.append(username)
1552 subprocess.check_call(cmd)
1553 user_info = pwd.getpwnam(username)
1554 return user_info
1555
1556
1557+def user_exists(username):
1558+ """Check if a user exists"""
1559+ try:
1560+ pwd.getpwnam(username)
1561+ user_exists = True
1562+ except KeyError:
1563+ user_exists = False
1564+ return user_exists
1565+
1566+
1567+def uid_exists(uid):
1568+ """Check if a uid exists"""
1569+ try:
1570+ pwd.getpwuid(uid)
1571+ uid_exists = True
1572+ except KeyError:
1573+ uid_exists = False
1574+ return uid_exists
1575+
1576+
1577+def group_exists(groupname):
1578+ """Check if a group exists"""
1579+ try:
1580+ grp.getgrnam(groupname)
1581+ group_exists = True
1582+ except KeyError:
1583+ group_exists = False
1584+ return group_exists
1585+
1586+
1587+def gid_exists(gid):
1588+ """Check if a gid exists"""
1589+ try:
1590+ grp.getgrgid(gid)
1591+ gid_exists = True
1592+ except KeyError:
1593+ gid_exists = False
1594+ return gid_exists
1595+
1596+
1597+def add_group(group_name, system_group=False, gid=None):
1598+ """Add a group to the system
1599+
1600+ Will log but otherwise succeed if the group already exists.
1601+
1602+ :param str group_name: group to create
1603+ :param bool system_group: Create system group
1604+ :param int gid: GID for user being created
1605+
1606+ :returns: The password database entry struct, as returned by `grp.getgrnam`
1607+ """
1608+ try:
1609+ group_info = grp.getgrnam(group_name)
1610+ log('group {0} already exists!'.format(group_name))
1611+ if gid:
1612+ group_info = grp.getgrgid(gid)
1613+ log('group with gid {0} already exists!'.format(gid))
1614+ except KeyError:
1615+ log('creating group {0}'.format(group_name))
1616+ cmd = ['addgroup']
1617+ if gid:
1618+ cmd.extend(['--gid', str(gid)])
1619+ if system_group:
1620+ cmd.append('--system')
1621+ else:
1622+ cmd.extend([
1623+ '--group',
1624+ ])
1625+ cmd.append(group_name)
1626+ subprocess.check_call(cmd)
1627+ group_info = grp.getgrnam(group_name)
1628+ return group_info
1629+
1630+
1631 def add_user_to_group(username, group):
1632 """Add a user to a group"""
1633- cmd = [
1634- 'gpasswd', '-a',
1635- username,
1636- group
1637- ]
1638+ cmd = ['gpasswd', '-a', username, group]
1639 log("Adding user {} to group {}".format(username, group))
1640 subprocess.check_call(cmd)
1641
1642@@ -142,35 +337,36 @@
1643 uid = pwd.getpwnam(owner).pw_uid
1644 gid = grp.getgrnam(group).gr_gid
1645 realpath = os.path.abspath(path)
1646- if os.path.exists(realpath):
1647- if force and not os.path.isdir(realpath):
1648+ path_exists = os.path.exists(realpath)
1649+ if path_exists and force:
1650+ if not os.path.isdir(realpath):
1651 log("Removing non-directory file {} prior to mkdir()".format(path))
1652 os.unlink(realpath)
1653- else:
1654+ os.makedirs(realpath, perms)
1655+ elif not path_exists:
1656 os.makedirs(realpath, perms)
1657 os.chown(realpath, uid, gid)
1658+ os.chmod(realpath, perms)
1659
1660
1661 def write_file(path, content, owner='root', group='root', perms=0o444):
1662- """Create or overwrite a file with the contents of a string"""
1663+ """Create or overwrite a file with the contents of a byte string."""
1664 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
1665 uid = pwd.getpwnam(owner).pw_uid
1666 gid = grp.getgrnam(group).gr_gid
1667- with open(path, 'w') as target:
1668+ with open(path, 'wb') as target:
1669 os.fchown(target.fileno(), uid, gid)
1670 os.fchmod(target.fileno(), perms)
1671 target.write(content)
1672
1673
1674 def fstab_remove(mp):
1675- """Remove the given mountpoint entry from /etc/fstab
1676- """
1677+ """Remove the given mountpoint entry from /etc/fstab"""
1678 return Fstab.remove_by_mountpoint(mp)
1679
1680
1681 def fstab_add(dev, mp, fs, options=None):
1682- """Adds the given device entry to the /etc/fstab file
1683- """
1684+ """Adds the given device entry to the /etc/fstab file"""
1685 return Fstab.add(dev, mp, fs, options=options)
1686
1687
1688@@ -214,9 +410,19 @@
1689 return system_mounts
1690
1691
1692+def fstab_mount(mountpoint):
1693+ """Mount filesystem using fstab"""
1694+ cmd_args = ['mount', mountpoint]
1695+ try:
1696+ subprocess.check_output(cmd_args)
1697+ except subprocess.CalledProcessError as e:
1698+ log('Error unmounting {}\n{}'.format(mountpoint, e.output))
1699+ return False
1700+ return True
1701+
1702+
1703 def file_hash(path, hash_type='md5'):
1704- """
1705- Generate a hash checksum of the contents of 'path' or None if not found.
1706+ """Generate a hash checksum of the contents of 'path' or None if not found.
1707
1708 :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
1709 such as md5, sha1, sha256, sha512, etc.
1710@@ -230,9 +436,22 @@
1711 return None
1712
1713
1714+def path_hash(path):
1715+ """Generate a hash checksum of all files matching 'path'. Standard
1716+ wildcards like '*' and '?' are supported, see documentation for the 'glob'
1717+ module for more information.
1718+
1719+ :return: dict: A { filename: hash } dictionary for all matched files.
1720+ Empty if none found.
1721+ """
1722+ return {
1723+ filename: file_hash(filename)
1724+ for filename in glob.iglob(path)
1725+ }
1726+
1727+
1728 def check_hash(path, checksum, hash_type='md5'):
1729- """
1730- Validate a file using a cryptographic checksum.
1731+ """Validate a file using a cryptographic checksum.
1732
1733 :param str checksum: Value of the checksum used to validate the file.
1734 :param str hash_type: Hash algorithm used to generate `checksum`.
1735@@ -247,46 +466,80 @@
1736
1737
1738 class ChecksumError(ValueError):
1739+ """A class derived from Value error to indicate the checksum failed."""
1740 pass
1741
1742
1743-def restart_on_change(restart_map, stopstart=False):
1744+def restart_on_change(restart_map, stopstart=False, restart_functions=None):
1745 """Restart services based on configuration files changing
1746
1747 This function is used a decorator, for example::
1748
1749 @restart_on_change({
1750 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
1751+ '/etc/apache/sites-enabled/*': [ 'apache2' ]
1752 })
1753- def ceph_client_changed():
1754+ def config_changed():
1755 pass # your code here
1756
1757 In this example, the cinder-api and cinder-volume services
1758 would be restarted if /etc/ceph/ceph.conf is changed by the
1759- ceph_client_changed function.
1760+ ceph_client_changed function. The apache2 service would be
1761+ restarted if any file matching the pattern got changed, created
1762+ or removed. Standard wildcards are supported, see documentation
1763+ for the 'glob' module for more information.
1764+
1765+ @param restart_map: {path_file_name: [service_name, ...]
1766+ @param stopstart: DEFAULT false; whether to stop, start OR restart
1767+ @param restart_functions: nonstandard functions to use to restart services
1768+ {svc: func, ...}
1769+ @returns result from decorated function
1770 """
1771 def wrap(f):
1772- def wrapped_f(*args):
1773- checksums = {}
1774- for path in restart_map:
1775- checksums[path] = file_hash(path)
1776- f(*args)
1777- restarts = []
1778- for path in restart_map:
1779- if checksums[path] != file_hash(path):
1780- restarts += restart_map[path]
1781- services_list = list(OrderedDict.fromkeys(restarts))
1782- if not stopstart:
1783- for service_name in services_list:
1784- service('restart', service_name)
1785- else:
1786- for action in ['stop', 'start']:
1787- for service_name in services_list:
1788- service(action, service_name)
1789+ @functools.wraps(f)
1790+ def wrapped_f(*args, **kwargs):
1791+ return restart_on_change_helper(
1792+ (lambda: f(*args, **kwargs)), restart_map, stopstart,
1793+ restart_functions)
1794 return wrapped_f
1795 return wrap
1796
1797
1798+def restart_on_change_helper(lambda_f, restart_map, stopstart=False,
1799+ restart_functions=None):
1800+ """Helper function to perform the restart_on_change function.
1801+
1802+ This is provided for decorators to restart services if files described
1803+ in the restart_map have changed after an invocation of lambda_f().
1804+
1805+ @param lambda_f: function to call.
1806+ @param restart_map: {file: [service, ...]}
1807+ @param stopstart: whether to stop, start or restart a service
1808+ @param restart_functions: nonstandard functions to use to restart services
1809+ {svc: func, ...}
1810+ @returns result of lambda_f()
1811+ """
1812+ if restart_functions is None:
1813+ restart_functions = {}
1814+ checksums = {path: path_hash(path) for path in restart_map}
1815+ r = lambda_f()
1816+ # create a list of lists of the services to restart
1817+ restarts = [restart_map[path]
1818+ for path in restart_map
1819+ if path_hash(path) != checksums[path]]
1820+ # create a flat list of ordered services without duplicates from lists
1821+ services_list = list(OrderedDict.fromkeys(itertools.chain(*restarts)))
1822+ if services_list:
1823+ actions = ('stop', 'start') if stopstart else ('restart',)
1824+ for service_name in services_list:
1825+ if service_name in restart_functions:
1826+ restart_functions[service_name](service_name)
1827+ else:
1828+ for action in actions:
1829+ service(action, service_name)
1830+ return r
1831+
1832+
1833 def lsb_release():
1834 """Return /etc/lsb-release in a dict"""
1835 d = {}
1836@@ -300,45 +553,105 @@
1837 def pwgen(length=None):
1838 """Generate a random pasword."""
1839 if length is None:
1840+ # A random length is ok to use a weak PRNG
1841 length = random.choice(range(35, 45))
1842 alphanumeric_chars = [
1843 l for l in (string.ascii_letters + string.digits)
1844 if l not in 'l0QD1vAEIOUaeiou']
1845+ # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
1846+ # actual password
1847+ random_generator = random.SystemRandom()
1848 random_chars = [
1849- random.choice(alphanumeric_chars) for _ in range(length)]
1850+ random_generator.choice(alphanumeric_chars) for _ in range(length)]
1851 return(''.join(random_chars))
1852
1853
1854-def list_nics(nic_type):
1855- '''Return a list of nics of given type(s)'''
1856+def is_phy_iface(interface):
1857+ """Returns True if interface is not virtual, otherwise False."""
1858+ if interface:
1859+ sys_net = '/sys/class/net'
1860+ if os.path.isdir(sys_net):
1861+ for iface in glob.glob(os.path.join(sys_net, '*')):
1862+ if '/virtual/' in os.path.realpath(iface):
1863+ continue
1864+
1865+ if interface == os.path.basename(iface):
1866+ return True
1867+
1868+ return False
1869+
1870+
1871+def get_bond_master(interface):
1872+ """Returns bond master if interface is bond slave otherwise None.
1873+
1874+ NOTE: the provided interface is expected to be physical
1875+ """
1876+ if interface:
1877+ iface_path = '/sys/class/net/%s' % (interface)
1878+ if os.path.exists(iface_path):
1879+ if '/virtual/' in os.path.realpath(iface_path):
1880+ return None
1881+
1882+ master = os.path.join(iface_path, 'master')
1883+ if os.path.exists(master):
1884+ master = os.path.realpath(master)
1885+ # make sure it is a bond master
1886+ if os.path.exists(os.path.join(master, 'bonding')):
1887+ return os.path.basename(master)
1888+
1889+ return None
1890+
1891+
1892+def list_nics(nic_type=None):
1893+ """Return a list of nics of given type(s)"""
1894 if isinstance(nic_type, six.string_types):
1895 int_types = [nic_type]
1896 else:
1897 int_types = nic_type
1898+
1899 interfaces = []
1900- for int_type in int_types:
1901- cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
1902+ if nic_type:
1903+ for int_type in int_types:
1904+ cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
1905+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
1906+ ip_output = ip_output.split('\n')
1907+ ip_output = (line for line in ip_output if line)
1908+ for line in ip_output:
1909+ if line.split()[1].startswith(int_type):
1910+ matched = re.search('.*: (' + int_type +
1911+ r'[0-9]+\.[0-9]+)@.*', line)
1912+ if matched:
1913+ iface = matched.groups()[0]
1914+ else:
1915+ iface = line.split()[1].replace(":", "")
1916+
1917+ if iface not in interfaces:
1918+ interfaces.append(iface)
1919+ else:
1920+ cmd = ['ip', 'a']
1921 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1922- ip_output = (line for line in ip_output if line)
1923+ ip_output = (line.strip() for line in ip_output if line)
1924+
1925+ key = re.compile('^[0-9]+:\s+(.+):')
1926 for line in ip_output:
1927- if line.split()[1].startswith(int_type):
1928- matched = re.search('.*: (bond[0-9]+\.[0-9]+)@.*', line)
1929- if matched:
1930- interface = matched.groups()[0]
1931- else:
1932- interface = line.split()[1].replace(":", "")
1933- interfaces.append(interface)
1934+ matched = re.search(key, line)
1935+ if matched:
1936+ iface = matched.group(1)
1937+ iface = iface.partition("@")[0]
1938+ if iface not in interfaces:
1939+ interfaces.append(iface)
1940
1941 return interfaces
1942
1943
1944 def set_nic_mtu(nic, mtu):
1945- '''Set MTU on a network interface'''
1946+ """Set the Maximum Transmission Unit (MTU) on a network interface."""
1947 cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]
1948 subprocess.check_call(cmd)
1949
1950
1951 def get_nic_mtu(nic):
1952+ """Return the Maximum Transmission Unit (MTU) for a network interface."""
1953 cmd = ['ip', 'addr', 'show', nic]
1954 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
1955 mtu = ""
1956@@ -350,6 +663,7 @@
1957
1958
1959 def get_nic_hwaddr(nic):
1960+ """Return the Media Access Control (MAC) for a network interface."""
1961 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
1962 ip_output = subprocess.check_output(cmd).decode('UTF-8')
1963 hwaddr = ""
1964@@ -360,37 +674,91 @@
1965
1966
1967 def cmp_pkgrevno(package, revno, pkgcache=None):
1968- '''Compare supplied revno with the revno of the installed package
1969+ """Compare supplied revno with the revno of the installed package
1970
1971 * 1 => Installed revno is greater than supplied arg
1972 * 0 => Installed revno is the same as supplied arg
1973 * -1 => Installed revno is less than supplied arg
1974
1975- '''
1976+ This function imports apt_cache function from charmhelpers.fetch if
1977+ the pkgcache argument is None. Be sure to add charmhelpers.fetch if
1978+ you call this function, or pass an apt_pkg.Cache() instance.
1979+ """
1980 import apt_pkg
1981- from charmhelpers.fetch import apt_cache
1982 if not pkgcache:
1983+ from charmhelpers.fetch import apt_cache
1984 pkgcache = apt_cache()
1985 pkg = pkgcache[package]
1986 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
1987
1988
1989 @contextmanager
1990-def chdir(d):
1991+def chdir(directory):
1992+ """Change the current working directory to a different directory for a code
1993+ block and return the previous directory after the block exits. Useful to
1994+ run commands from a specificed directory.
1995+
1996+ :param str directory: The directory path to change to for this context.
1997+ """
1998 cur = os.getcwd()
1999 try:
2000- yield os.chdir(d)
2001+ yield os.chdir(directory)
2002 finally:
2003 os.chdir(cur)
2004
2005
2006-def chownr(path, owner, group):
2007+def chownr(path, owner, group, follow_links=True, chowntopdir=False):
2008+ """Recursively change user and group ownership of files and directories
2009+ in given path. Doesn't chown path itself by default, only its children.
2010+
2011+ :param str path: The string path to start changing ownership.
2012+ :param str owner: The owner string to use when looking up the uid.
2013+ :param str group: The group string to use when looking up the gid.
2014+ :param bool follow_links: Also Chown links if True
2015+ :param bool chowntopdir: Also chown path itself if True
2016+ """
2017 uid = pwd.getpwnam(owner).pw_uid
2018 gid = grp.getgrnam(group).gr_gid
2019+ if follow_links:
2020+ chown = os.chown
2021+ else:
2022+ chown = os.lchown
2023
2024+ if chowntopdir:
2025+ broken_symlink = os.path.lexists(path) and not os.path.exists(path)
2026+ if not broken_symlink:
2027+ chown(path, uid, gid)
2028 for root, dirs, files in os.walk(path):
2029 for name in dirs + files:
2030 full = os.path.join(root, name)
2031 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
2032 if not broken_symlink:
2033- os.chown(full, uid, gid)
2034+ chown(full, uid, gid)
2035+
2036+
2037+def lchownr(path, owner, group):
2038+ """Recursively change user and group ownership of files and directories
2039+ in a given path, not following symbolic links. See the documentation for
2040+ 'os.lchown' for more information.
2041+
2042+ :param str path: The string path to start changing ownership.
2043+ :param str owner: The owner string to use when looking up the uid.
2044+ :param str group: The group string to use when looking up the gid.
2045+ """
2046+ chownr(path, owner, group, follow_links=False)
2047+
2048+
2049+def get_total_ram():
2050+ """The total amount of system RAM in bytes.
2051+
2052+ This is what is reported by the OS, and may be overcommitted when
2053+ there are multiple containers hosted on the same machine.
2054+ """
2055+ with open('/proc/meminfo', 'r') as f:
2056+ for line in f.readlines():
2057+ if line:
2058+ key, value, unit = line.split()
2059+ if key == 'MemTotal:':
2060+ assert unit == 'kB', 'Unknown unit'
2061+ return int(value) * 1024 # Classic, not KiB.
2062+ raise NotImplementedError()
2063
2064=== added file 'lib/charmhelpers/core/hugepage.py'
2065--- lib/charmhelpers/core/hugepage.py 1970-01-01 00:00:00 +0000
2066+++ lib/charmhelpers/core/hugepage.py 2016-07-12 15:38:59 +0000
2067@@ -0,0 +1,71 @@
2068+# -*- coding: utf-8 -*-
2069+
2070+# Copyright 2014-2015 Canonical Limited.
2071+#
2072+# This file is part of charm-helpers.
2073+#
2074+# charm-helpers is free software: you can redistribute it and/or modify
2075+# it under the terms of the GNU Lesser General Public License version 3 as
2076+# published by the Free Software Foundation.
2077+#
2078+# charm-helpers is distributed in the hope that it will be useful,
2079+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2080+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2081+# GNU Lesser General Public License for more details.
2082+#
2083+# You should have received a copy of the GNU Lesser General Public License
2084+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2085+
2086+import yaml
2087+from charmhelpers.core import fstab
2088+from charmhelpers.core import sysctl
2089+from charmhelpers.core.host import (
2090+ add_group,
2091+ add_user_to_group,
2092+ fstab_mount,
2093+ mkdir,
2094+)
2095+from charmhelpers.core.strutils import bytes_from_string
2096+from subprocess import check_output
2097+
2098+
2099+def hugepage_support(user, group='hugetlb', nr_hugepages=256,
2100+ max_map_count=65536, mnt_point='/run/hugepages/kvm',
2101+ pagesize='2MB', mount=True, set_shmmax=False):
2102+ """Enable hugepages on system.
2103+
2104+ Args:
2105+ user (str) -- Username to allow access to hugepages to
2106+ group (str) -- Group name to own hugepages
2107+ nr_hugepages (int) -- Number of pages to reserve
2108+ max_map_count (int) -- Number of Virtual Memory Areas a process can own
2109+ mnt_point (str) -- Directory to mount hugepages on
2110+ pagesize (str) -- Size of hugepages
2111+ mount (bool) -- Whether to Mount hugepages
2112+ """
2113+ group_info = add_group(group)
2114+ gid = group_info.gr_gid
2115+ add_user_to_group(user, group)
2116+ if max_map_count < 2 * nr_hugepages:
2117+ max_map_count = 2 * nr_hugepages
2118+ sysctl_settings = {
2119+ 'vm.nr_hugepages': nr_hugepages,
2120+ 'vm.max_map_count': max_map_count,
2121+ 'vm.hugetlb_shm_group': gid,
2122+ }
2123+ if set_shmmax:
2124+ shmmax_current = int(check_output(['sysctl', '-n', 'kernel.shmmax']))
2125+ shmmax_minsize = bytes_from_string(pagesize) * nr_hugepages
2126+ if shmmax_minsize > shmmax_current:
2127+ sysctl_settings['kernel.shmmax'] = shmmax_minsize
2128+ sysctl.create(yaml.dump(sysctl_settings), '/etc/sysctl.d/10-hugepage.conf')
2129+ mkdir(mnt_point, owner='root', group='root', perms=0o755, force=False)
2130+ lfstab = fstab.Fstab()
2131+ fstab_entry = lfstab.get_entry_by_attr('mountpoint', mnt_point)
2132+ if fstab_entry:
2133+ lfstab.remove_entry(fstab_entry)
2134+ entry = lfstab.Entry('nodev', mnt_point, 'hugetlbfs',
2135+ 'mode=1770,gid={},pagesize={}'.format(gid, pagesize), 0, 0)
2136+ lfstab.add_entry(entry)
2137+ if mount:
2138+ fstab_mount(mnt_point)
2139
2140=== added file 'lib/charmhelpers/core/kernel.py'
2141--- lib/charmhelpers/core/kernel.py 1970-01-01 00:00:00 +0000
2142+++ lib/charmhelpers/core/kernel.py 2016-07-12 15:38:59 +0000
2143@@ -0,0 +1,68 @@
2144+#!/usr/bin/env python
2145+# -*- coding: utf-8 -*-
2146+
2147+# Copyright 2014-2015 Canonical Limited.
2148+#
2149+# This file is part of charm-helpers.
2150+#
2151+# charm-helpers is free software: you can redistribute it and/or modify
2152+# it under the terms of the GNU Lesser General Public License version 3 as
2153+# published by the Free Software Foundation.
2154+#
2155+# charm-helpers is distributed in the hope that it will be useful,
2156+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2157+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2158+# GNU Lesser General Public License for more details.
2159+#
2160+# You should have received a copy of the GNU Lesser General Public License
2161+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2162+
2163+__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
2164+
2165+from charmhelpers.core.hookenv import (
2166+ log,
2167+ INFO
2168+)
2169+
2170+from subprocess import check_call, check_output
2171+import re
2172+
2173+
2174+def modprobe(module, persist=True):
2175+ """Load a kernel module and configure for auto-load on reboot."""
2176+ cmd = ['modprobe', module]
2177+
2178+ log('Loading kernel module %s' % module, level=INFO)
2179+
2180+ check_call(cmd)
2181+ if persist:
2182+ with open('/etc/modules', 'r+') as modules:
2183+ if module not in modules.read():
2184+ modules.write(module)
2185+
2186+
2187+def rmmod(module, force=False):
2188+ """Remove a module from the linux kernel"""
2189+ cmd = ['rmmod']
2190+ if force:
2191+ cmd.append('-f')
2192+ cmd.append(module)
2193+ log('Removing kernel module %s' % module, level=INFO)
2194+ return check_call(cmd)
2195+
2196+
2197+def lsmod():
2198+ """Shows what kernel modules are currently loaded"""
2199+ return check_output(['lsmod'],
2200+ universal_newlines=True)
2201+
2202+
2203+def is_module_loaded(module):
2204+ """Checks if a kernel module is already loaded"""
2205+ matches = re.findall('^%s[ ]+' % module, lsmod(), re.M)
2206+ return len(matches) > 0
2207+
2208+
2209+def update_initramfs(version='all'):
2210+ """Updates an initramfs image"""
2211+ return check_call(["update-initramfs", "-k", version, "-u"])
2212
2213=== modified file 'lib/charmhelpers/core/services/__init__.py'
2214--- lib/charmhelpers/core/services/__init__.py 2014-12-02 19:35:26 +0000
2215+++ lib/charmhelpers/core/services/__init__.py 2016-07-12 15:38:59 +0000
2216@@ -1,2 +1,18 @@
2217+# Copyright 2014-2015 Canonical Limited.
2218+#
2219+# This file is part of charm-helpers.
2220+#
2221+# charm-helpers is free software: you can redistribute it and/or modify
2222+# it under the terms of the GNU Lesser General Public License version 3 as
2223+# published by the Free Software Foundation.
2224+#
2225+# charm-helpers is distributed in the hope that it will be useful,
2226+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2227+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2228+# GNU Lesser General Public License for more details.
2229+#
2230+# You should have received a copy of the GNU Lesser General Public License
2231+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2232+
2233 from .base import * # NOQA
2234 from .helpers import * # NOQA
2235
2236=== modified file 'lib/charmhelpers/core/services/base.py'
2237--- lib/charmhelpers/core/services/base.py 2014-12-02 19:35:26 +0000
2238+++ lib/charmhelpers/core/services/base.py 2016-07-12 15:38:59 +0000
2239@@ -1,7 +1,23 @@
2240+# Copyright 2014-2015 Canonical Limited.
2241+#
2242+# This file is part of charm-helpers.
2243+#
2244+# charm-helpers is free software: you can redistribute it and/or modify
2245+# it under the terms of the GNU Lesser General Public License version 3 as
2246+# published by the Free Software Foundation.
2247+#
2248+# charm-helpers is distributed in the hope that it will be useful,
2249+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2250+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2251+# GNU Lesser General Public License for more details.
2252+#
2253+# You should have received a copy of the GNU Lesser General Public License
2254+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2255+
2256 import os
2257-import re
2258 import json
2259-from collections import Iterable
2260+from inspect import getargspec
2261+from collections import Iterable, OrderedDict
2262
2263 from charmhelpers.core import host
2264 from charmhelpers.core import hookenv
2265@@ -103,7 +119,7 @@
2266 """
2267 self._ready_file = os.path.join(hookenv.charm_dir(), 'READY-SERVICES.json')
2268 self._ready = None
2269- self.services = {}
2270+ self.services = OrderedDict()
2271 for service in services or []:
2272 service_name = service['service']
2273 self.services[service_name] = service
2274@@ -112,15 +128,18 @@
2275 """
2276 Handle the current hook by doing The Right Thing with the registered services.
2277 """
2278- hook_name = hookenv.hook_name()
2279- if hook_name == 'stop':
2280- self.stop_services()
2281- else:
2282- self.provide_data()
2283- self.reconfigure_services()
2284- cfg = hookenv.config()
2285- if cfg.implicit_save:
2286- cfg.save()
2287+ hookenv._run_atstart()
2288+ try:
2289+ hook_name = hookenv.hook_name()
2290+ if hook_name == 'stop':
2291+ self.stop_services()
2292+ else:
2293+ self.reconfigure_services()
2294+ self.provide_data()
2295+ except SystemExit as x:
2296+ if x.code is None or x.code == 0:
2297+ hookenv._run_atexit()
2298+ hookenv._run_atexit()
2299
2300 def provide_data(self):
2301 """
2302@@ -129,15 +148,36 @@
2303 A provider must have a `name` attribute, which indicates which relation
2304 to set data on, and a `provide_data()` method, which returns a dict of
2305 data to set.
2306+
2307+ The `provide_data()` method can optionally accept two parameters:
2308+
2309+ * ``remote_service`` The name of the remote service that the data will
2310+ be provided to. The `provide_data()` method will be called once
2311+ for each connected service (not unit). This allows the method to
2312+ tailor its data to the given service.
2313+ * ``service_ready`` Whether or not the service definition had all of
2314+ its requirements met, and thus the ``data_ready`` callbacks run.
2315+
2316+ Note that the ``provided_data`` methods are now called **after** the
2317+ ``data_ready`` callbacks are run. This gives the ``data_ready`` callbacks
2318+ a chance to generate any data necessary for the providing to the remote
2319+ services.
2320 """
2321- hook_name = hookenv.hook_name()
2322- for service in self.services.values():
2323+ for service_name, service in self.services.items():
2324+ service_ready = self.is_ready(service_name)
2325 for provider in service.get('provided_data', []):
2326- if re.match(r'{}-relation-(joined|changed)'.format(provider.name), hook_name):
2327- data = provider.provide_data()
2328- _ready = provider._is_ready(data) if hasattr(provider, '_is_ready') else data
2329- if _ready:
2330- hookenv.relation_set(None, data)
2331+ for relid in hookenv.relation_ids(provider.name):
2332+ units = hookenv.related_units(relid)
2333+ if not units:
2334+ continue
2335+ remote_service = units[0].split('/')[0]
2336+ argspec = getargspec(provider.provide_data)
2337+ if len(argspec.args) > 1:
2338+ data = provider.provide_data(remote_service, service_ready)
2339+ else:
2340+ data = provider.provide_data()
2341+ if data:
2342+ hookenv.relation_set(relid, data)
2343
2344 def reconfigure_services(self, *service_names):
2345 """
2346
2347=== modified file 'lib/charmhelpers/core/services/helpers.py'
2348--- lib/charmhelpers/core/services/helpers.py 2014-12-02 19:35:26 +0000
2349+++ lib/charmhelpers/core/services/helpers.py 2016-07-12 15:38:59 +0000
2350@@ -1,6 +1,24 @@
2351+# Copyright 2014-2015 Canonical Limited.
2352+#
2353+# This file is part of charm-helpers.
2354+#
2355+# charm-helpers is free software: you can redistribute it and/or modify
2356+# it under the terms of the GNU Lesser General Public License version 3 as
2357+# published by the Free Software Foundation.
2358+#
2359+# charm-helpers is distributed in the hope that it will be useful,
2360+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2361+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2362+# GNU Lesser General Public License for more details.
2363+#
2364+# You should have received a copy of the GNU Lesser General Public License
2365+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2366+
2367 import os
2368 import yaml
2369+
2370 from charmhelpers.core import hookenv
2371+from charmhelpers.core import host
2372 from charmhelpers.core import templating
2373
2374 from charmhelpers.core.services.base import ManagerCallback
2375@@ -29,12 +47,14 @@
2376 """
2377 name = None
2378 interface = None
2379- required_keys = []
2380
2381 def __init__(self, name=None, additional_required_keys=None):
2382+ if not hasattr(self, 'required_keys'):
2383+ self.required_keys = []
2384+
2385 if name is not None:
2386 self.name = name
2387- if additional_required_keys is not None:
2388+ if additional_required_keys:
2389 self.required_keys.extend(additional_required_keys)
2390 self.get_data()
2391
2392@@ -118,7 +138,10 @@
2393 """
2394 name = 'db'
2395 interface = 'mysql'
2396- required_keys = ['host', 'user', 'password', 'database']
2397+
2398+ def __init__(self, *args, **kwargs):
2399+ self.required_keys = ['host', 'user', 'password', 'database']
2400+ RelationContext.__init__(self, *args, **kwargs)
2401
2402
2403 class HttpRelation(RelationContext):
2404@@ -130,7 +153,10 @@
2405 """
2406 name = 'website'
2407 interface = 'http'
2408- required_keys = ['host', 'port']
2409+
2410+ def __init__(self, *args, **kwargs):
2411+ self.required_keys = ['host', 'port']
2412+ RelationContext.__init__(self, *args, **kwargs)
2413
2414 def provide_data(self):
2415 return {
2416@@ -215,28 +241,51 @@
2417 action.
2418
2419 :param str source: The template source file, relative to
2420- `$CHARM_DIR/templates`
2421+ `$CHARM_DIR/templates`
2422
2423- :param str target: The target to write the rendered template to
2424+ :param str target: The target to write the rendered template to (or None)
2425 :param str owner: The owner of the rendered file
2426 :param str group: The group of the rendered file
2427 :param int perms: The permissions of the rendered file
2428+ :param partial on_change_action: functools partial to be executed when
2429+ rendered file changes
2430+ :param jinja2 loader template_loader: A jinja2 template loader
2431+
2432+ :return str: The rendered template
2433 """
2434 def __init__(self, source, target,
2435- owner='root', group='root', perms=0o444):
2436+ owner='root', group='root', perms=0o444,
2437+ on_change_action=None, template_loader=None):
2438 self.source = source
2439 self.target = target
2440 self.owner = owner
2441 self.group = group
2442 self.perms = perms
2443+ self.on_change_action = on_change_action
2444+ self.template_loader = template_loader
2445
2446 def __call__(self, manager, service_name, event_name):
2447+ pre_checksum = ''
2448+ if self.on_change_action and os.path.isfile(self.target):
2449+ pre_checksum = host.file_hash(self.target)
2450 service = manager.get_service(service_name)
2451- context = {}
2452+ context = {'ctx': {}}
2453 for ctx in service.get('required_data', []):
2454 context.update(ctx)
2455- templating.render(self.source, self.target, context,
2456- self.owner, self.group, self.perms)
2457+ context['ctx'].update(ctx)
2458+
2459+ result = templating.render(self.source, self.target, context,
2460+ self.owner, self.group, self.perms,
2461+ template_loader=self.template_loader)
2462+ if self.on_change_action:
2463+ if pre_checksum == host.file_hash(self.target):
2464+ hookenv.log(
2465+ 'No change detected: {}'.format(self.target),
2466+ hookenv.DEBUG)
2467+ else:
2468+ self.on_change_action()
2469+
2470+ return result
2471
2472
2473 # Convenience aliases for templates
2474
2475=== added file 'lib/charmhelpers/core/strutils.py'
2476--- lib/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000
2477+++ lib/charmhelpers/core/strutils.py 2016-07-12 15:38:59 +0000
2478@@ -0,0 +1,72 @@
2479+#!/usr/bin/env python
2480+# -*- coding: utf-8 -*-
2481+
2482+# Copyright 2014-2015 Canonical Limited.
2483+#
2484+# This file is part of charm-helpers.
2485+#
2486+# charm-helpers is free software: you can redistribute it and/or modify
2487+# it under the terms of the GNU Lesser General Public License version 3 as
2488+# published by the Free Software Foundation.
2489+#
2490+# charm-helpers is distributed in the hope that it will be useful,
2491+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2492+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2493+# GNU Lesser General Public License for more details.
2494+#
2495+# You should have received a copy of the GNU Lesser General Public License
2496+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2497+
2498+import six
2499+import re
2500+
2501+
2502+def bool_from_string(value):
2503+ """Interpret string value as boolean.
2504+
2505+ Returns True if value translates to True otherwise False.
2506+ """
2507+ if isinstance(value, six.string_types):
2508+ value = six.text_type(value)
2509+ else:
2510+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
2511+ raise ValueError(msg)
2512+
2513+ value = value.strip().lower()
2514+
2515+ if value in ['y', 'yes', 'true', 't', 'on']:
2516+ return True
2517+ elif value in ['n', 'no', 'false', 'f', 'off']:
2518+ return False
2519+
2520+ msg = "Unable to interpret string value '%s' as boolean" % (value)
2521+ raise ValueError(msg)
2522+
2523+
2524+def bytes_from_string(value):
2525+ """Interpret human readable string value as bytes.
2526+
2527+ Returns int
2528+ """
2529+ BYTE_POWER = {
2530+ 'K': 1,
2531+ 'KB': 1,
2532+ 'M': 2,
2533+ 'MB': 2,
2534+ 'G': 3,
2535+ 'GB': 3,
2536+ 'T': 4,
2537+ 'TB': 4,
2538+ 'P': 5,
2539+ 'PB': 5,
2540+ }
2541+ if isinstance(value, six.string_types):
2542+ value = six.text_type(value)
2543+ else:
2544+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
2545+ raise ValueError(msg)
2546+ matches = re.match("([0-9]+)([a-zA-Z]+)", value)
2547+ if not matches:
2548+ msg = "Unable to interpret string value '%s' as bytes" % (value)
2549+ raise ValueError(msg)
2550+ return int(matches.group(1)) * (1024 ** BYTE_POWER[matches.group(2)])
2551
2552=== modified file 'lib/charmhelpers/core/sysctl.py'
2553--- lib/charmhelpers/core/sysctl.py 2014-12-02 19:35:26 +0000
2554+++ lib/charmhelpers/core/sysctl.py 2016-07-12 15:38:59 +0000
2555@@ -1,7 +1,21 @@
2556 #!/usr/bin/env python
2557 # -*- coding: utf-8 -*-
2558
2559-__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
2560+# Copyright 2014-2015 Canonical Limited.
2561+#
2562+# This file is part of charm-helpers.
2563+#
2564+# charm-helpers is free software: you can redistribute it and/or modify
2565+# it under the terms of the GNU Lesser General Public License version 3 as
2566+# published by the Free Software Foundation.
2567+#
2568+# charm-helpers is distributed in the hope that it will be useful,
2569+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2570+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2571+# GNU Lesser General Public License for more details.
2572+#
2573+# You should have received a copy of the GNU Lesser General Public License
2574+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2575
2576 import yaml
2577
2578@@ -10,25 +24,33 @@
2579 from charmhelpers.core.hookenv import (
2580 log,
2581 DEBUG,
2582+ ERROR,
2583 )
2584
2585+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
2586+
2587
2588 def create(sysctl_dict, sysctl_file):
2589 """Creates a sysctl.conf file from a YAML associative array
2590
2591- :param sysctl_dict: a dict of sysctl options eg { 'kernel.max_pid': 1337 }
2592- :type sysctl_dict: dict
2593+ :param sysctl_dict: a YAML-formatted string of sysctl options eg "{ 'kernel.max_pid': 1337 }"
2594+ :type sysctl_dict: str
2595 :param sysctl_file: path to the sysctl file to be saved
2596 :type sysctl_file: str or unicode
2597 :returns: None
2598 """
2599- sysctl_dict = yaml.load(sysctl_dict)
2600+ try:
2601+ sysctl_dict_parsed = yaml.safe_load(sysctl_dict)
2602+ except yaml.YAMLError:
2603+ log("Error parsing YAML sysctl_dict: {}".format(sysctl_dict),
2604+ level=ERROR)
2605+ return
2606
2607 with open(sysctl_file, "w") as fd:
2608- for key, value in sysctl_dict.items():
2609+ for key, value in sysctl_dict_parsed.items():
2610 fd.write("{}={}\n".format(key, value))
2611
2612- log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict),
2613+ log("Updating sysctl_file: %s values: %s" % (sysctl_file, sysctl_dict_parsed),
2614 level=DEBUG)
2615
2616 check_call(["sysctl", "-p", sysctl_file])
2617
2618=== modified file 'lib/charmhelpers/core/templating.py'
2619--- lib/charmhelpers/core/templating.py 2014-12-02 19:35:26 +0000
2620+++ lib/charmhelpers/core/templating.py 2016-07-12 15:38:59 +0000
2621@@ -1,3 +1,19 @@
2622+# Copyright 2014-2015 Canonical Limited.
2623+#
2624+# This file is part of charm-helpers.
2625+#
2626+# charm-helpers is free software: you can redistribute it and/or modify
2627+# it under the terms of the GNU Lesser General Public License version 3 as
2628+# published by the Free Software Foundation.
2629+#
2630+# charm-helpers is distributed in the hope that it will be useful,
2631+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2632+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2633+# GNU Lesser General Public License for more details.
2634+#
2635+# You should have received a copy of the GNU Lesser General Public License
2636+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2637+
2638 import os
2639
2640 from charmhelpers.core import host
2641@@ -5,13 +21,14 @@
2642
2643
2644 def render(source, target, context, owner='root', group='root',
2645- perms=0o444, templates_dir=None):
2646+ perms=0o444, templates_dir=None, encoding='UTF-8', template_loader=None):
2647 """
2648 Render a template.
2649
2650 The `source` path, if not absolute, is relative to the `templates_dir`.
2651
2652- The `target` path should be absolute.
2653+ The `target` path should be absolute. It can also be `None`, in which
2654+ case no file will be written.
2655
2656 The context should be a dict containing the values to be replaced in the
2657 template.
2658@@ -20,6 +37,9 @@
2659
2660 If omitted, `templates_dir` defaults to the `templates` folder in the charm.
2661
2662+ The rendered template will be written to the file as well as being returned
2663+ as a string.
2664+
2665 Note: Using this requires python-jinja2; if it is not installed, calling
2666 this will attempt to use charmhelpers.fetch.apt_install to install it.
2667 """
2668@@ -36,17 +56,26 @@
2669 apt_install('python-jinja2', fatal=True)
2670 from jinja2 import FileSystemLoader, Environment, exceptions
2671
2672- if templates_dir is None:
2673- templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
2674- loader = Environment(loader=FileSystemLoader(templates_dir))
2675+ if template_loader:
2676+ template_env = Environment(loader=template_loader)
2677+ else:
2678+ if templates_dir is None:
2679+ templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
2680+ template_env = Environment(loader=FileSystemLoader(templates_dir))
2681 try:
2682 source = source
2683- template = loader.get_template(source)
2684+ template = template_env.get_template(source)
2685 except exceptions.TemplateNotFound as e:
2686 hookenv.log('Could not load template %s from %s.' %
2687 (source, templates_dir),
2688 level=hookenv.ERROR)
2689 raise e
2690 content = template.render(context)
2691- host.mkdir(os.path.dirname(target))
2692- host.write_file(target, content, owner, group, perms)
2693+ if target is not None:
2694+ target_dir = os.path.dirname(target)
2695+ if not os.path.exists(target_dir):
2696+ # This is a terrible default directory permission, as the file
2697+ # or its siblings will often contain secrets.
2698+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
2699+ host.write_file(target, content.encode(encoding), owner, group, perms)
2700+ return content
2701
2702=== added file 'lib/charmhelpers/core/unitdata.py'
2703--- lib/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000
2704+++ lib/charmhelpers/core/unitdata.py 2016-07-12 15:38:59 +0000
2705@@ -0,0 +1,521 @@
2706+#!/usr/bin/env python
2707+# -*- coding: utf-8 -*-
2708+#
2709+# Copyright 2014-2015 Canonical Limited.
2710+#
2711+# This file is part of charm-helpers.
2712+#
2713+# charm-helpers is free software: you can redistribute it and/or modify
2714+# it under the terms of the GNU Lesser General Public License version 3 as
2715+# published by the Free Software Foundation.
2716+#
2717+# charm-helpers is distributed in the hope that it will be useful,
2718+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2719+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2720+# GNU Lesser General Public License for more details.
2721+#
2722+# You should have received a copy of the GNU Lesser General Public License
2723+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2724+#
2725+#
2726+# Authors:
2727+# Kapil Thangavelu <kapil.foss@gmail.com>
2728+#
2729+"""
2730+Intro
2731+-----
2732+
2733+A simple way to store state in units. This provides a key value
2734+storage with support for versioned, transactional operation,
2735+and can calculate deltas from previous values to simplify unit logic
2736+when processing changes.
2737+
2738+
2739+Hook Integration
2740+----------------
2741+
2742+There are several extant frameworks for hook execution, including
2743+
2744+ - charmhelpers.core.hookenv.Hooks
2745+ - charmhelpers.core.services.ServiceManager
2746+
2747+The storage classes are framework agnostic, one simple integration is
2748+via the HookData contextmanager. It will record the current hook
2749+execution environment (including relation data, config data, etc.),
2750+setup a transaction and allow easy access to the changes from
2751+previously seen values. One consequence of the integration is the
2752+reservation of particular keys ('rels', 'unit', 'env', 'config',
2753+'charm_revisions') for their respective values.
2754+
2755+Here's a fully worked integration example using hookenv.Hooks::
2756+
2757+ from charmhelper.core import hookenv, unitdata
2758+
2759+ hook_data = unitdata.HookData()
2760+ db = unitdata.kv()
2761+ hooks = hookenv.Hooks()
2762+
2763+ @hooks.hook
2764+ def config_changed():
2765+ # Print all changes to configuration from previously seen
2766+ # values.
2767+ for changed, (prev, cur) in hook_data.conf.items():
2768+ print('config changed', changed,
2769+ 'previous value', prev,
2770+ 'current value', cur)
2771+
2772+ # Get some unit specific bookeeping
2773+ if not db.get('pkg_key'):
2774+ key = urllib.urlopen('https://example.com/pkg_key').read()
2775+ db.set('pkg_key', key)
2776+
2777+ # Directly access all charm config as a mapping.
2778+ conf = db.getrange('config', True)
2779+
2780+ # Directly access all relation data as a mapping
2781+ rels = db.getrange('rels', True)
2782+
2783+ if __name__ == '__main__':
2784+ with hook_data():
2785+ hook.execute()
2786+
2787+
2788+A more basic integration is via the hook_scope context manager which simply
2789+manages transaction scope (and records hook name, and timestamp)::
2790+
2791+ >>> from unitdata import kv
2792+ >>> db = kv()
2793+ >>> with db.hook_scope('install'):
2794+ ... # do work, in transactional scope.
2795+ ... db.set('x', 1)
2796+ >>> db.get('x')
2797+ 1
2798+
2799+
2800+Usage
2801+-----
2802+
2803+Values are automatically json de/serialized to preserve basic typing
2804+and complex data struct capabilities (dicts, lists, ints, booleans, etc).
2805+
2806+Individual values can be manipulated via get/set::
2807+
2808+ >>> kv.set('y', True)
2809+ >>> kv.get('y')
2810+ True
2811+
2812+ # We can set complex values (dicts, lists) as a single key.
2813+ >>> kv.set('config', {'a': 1, 'b': True'})
2814+
2815+ # Also supports returning dictionaries as a record which
2816+ # provides attribute access.
2817+ >>> config = kv.get('config', record=True)
2818+ >>> config.b
2819+ True
2820+
2821+
2822+Groups of keys can be manipulated with update/getrange::
2823+
2824+ >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
2825+ >>> kv.getrange('gui.', strip=True)
2826+ {'z': 1, 'y': 2}
2827+
2828+When updating values, its very helpful to understand which values
2829+have actually changed and how have they changed. The storage
2830+provides a delta method to provide for this::
2831+
2832+ >>> data = {'debug': True, 'option': 2}
2833+ >>> delta = kv.delta(data, 'config.')
2834+ >>> delta.debug.previous
2835+ None
2836+ >>> delta.debug.current
2837+ True
2838+ >>> delta
2839+ {'debug': (None, True), 'option': (None, 2)}
2840+
2841+Note the delta method does not persist the actual change, it needs to
2842+be explicitly saved via 'update' method::
2843+
2844+ >>> kv.update(data, 'config.')
2845+
2846+Values modified in the context of a hook scope retain historical values
2847+associated to the hookname.
2848+
2849+ >>> with db.hook_scope('config-changed'):
2850+ ... db.set('x', 42)
2851+ >>> db.gethistory('x')
2852+ [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
2853+ (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
2854+
2855+"""
2856+
2857+import collections
2858+import contextlib
2859+import datetime
2860+import itertools
2861+import json
2862+import os
2863+import pprint
2864+import sqlite3
2865+import sys
2866+
2867+__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
2868+
2869+
2870+class Storage(object):
2871+ """Simple key value database for local unit state within charms.
2872+
2873+ Modifications are not persisted unless :meth:`flush` is called.
2874+
2875+ To support dicts, lists, integer, floats, and booleans values
2876+ are automatically json encoded/decoded.
2877+ """
2878+ def __init__(self, path=None):
2879+ self.db_path = path
2880+ if path is None:
2881+ if 'UNIT_STATE_DB' in os.environ:
2882+ self.db_path = os.environ['UNIT_STATE_DB']
2883+ else:
2884+ self.db_path = os.path.join(
2885+ os.environ.get('CHARM_DIR', ''), '.unit-state.db')
2886+ self.conn = sqlite3.connect('%s' % self.db_path)
2887+ self.cursor = self.conn.cursor()
2888+ self.revision = None
2889+ self._closed = False
2890+ self._init()
2891+
2892+ def close(self):
2893+ if self._closed:
2894+ return
2895+ self.flush(False)
2896+ self.cursor.close()
2897+ self.conn.close()
2898+ self._closed = True
2899+
2900+ def get(self, key, default=None, record=False):
2901+ self.cursor.execute('select data from kv where key=?', [key])
2902+ result = self.cursor.fetchone()
2903+ if not result:
2904+ return default
2905+ if record:
2906+ return Record(json.loads(result[0]))
2907+ return json.loads(result[0])
2908+
2909+ def getrange(self, key_prefix, strip=False):
2910+ """
2911+ Get a range of keys starting with a common prefix as a mapping of
2912+ keys to values.
2913+
2914+ :param str key_prefix: Common prefix among all keys
2915+ :param bool strip: Optionally strip the common prefix from the key
2916+ names in the returned dict
2917+ :return dict: A (possibly empty) dict of key-value mappings
2918+ """
2919+ self.cursor.execute("select key, data from kv where key like ?",
2920+ ['%s%%' % key_prefix])
2921+ result = self.cursor.fetchall()
2922+
2923+ if not result:
2924+ return {}
2925+ if not strip:
2926+ key_prefix = ''
2927+ return dict([
2928+ (k[len(key_prefix):], json.loads(v)) for k, v in result])
2929+
2930+ def update(self, mapping, prefix=""):
2931+ """
2932+ Set the values of multiple keys at once.
2933+
2934+ :param dict mapping: Mapping of keys to values
2935+ :param str prefix: Optional prefix to apply to all keys in `mapping`
2936+ before setting
2937+ """
2938+ for k, v in mapping.items():
2939+ self.set("%s%s" % (prefix, k), v)
2940+
2941+ def unset(self, key):
2942+ """
2943+ Remove a key from the database entirely.
2944+ """
2945+ self.cursor.execute('delete from kv where key=?', [key])
2946+ if self.revision and self.cursor.rowcount:
2947+ self.cursor.execute(
2948+ 'insert into kv_revisions values (?, ?, ?)',
2949+ [key, self.revision, json.dumps('DELETED')])
2950+
2951+ def unsetrange(self, keys=None, prefix=""):
2952+ """
2953+ Remove a range of keys starting with a common prefix, from the database
2954+ entirely.
2955+
2956+ :param list keys: List of keys to remove.
2957+ :param str prefix: Optional prefix to apply to all keys in ``keys``
2958+ before removing.
2959+ """
2960+ if keys is not None:
2961+ keys = ['%s%s' % (prefix, key) for key in keys]
2962+ self.cursor.execute('delete from kv where key in (%s)' % ','.join(['?'] * len(keys)), keys)
2963+ if self.revision and self.cursor.rowcount:
2964+ self.cursor.execute(
2965+ 'insert into kv_revisions values %s' % ','.join(['(?, ?, ?)'] * len(keys)),
2966+ list(itertools.chain.from_iterable((key, self.revision, json.dumps('DELETED')) for key in keys)))
2967+ else:
2968+ self.cursor.execute('delete from kv where key like ?',
2969+ ['%s%%' % prefix])
2970+ if self.revision and self.cursor.rowcount:
2971+ self.cursor.execute(
2972+ 'insert into kv_revisions values (?, ?, ?)',
2973+ ['%s%%' % prefix, self.revision, json.dumps('DELETED')])
2974+
2975+ def set(self, key, value):
2976+ """
2977+ Set a value in the database.
2978+
2979+ :param str key: Key to set the value for
2980+ :param value: Any JSON-serializable value to be set
2981+ """
2982+ serialized = json.dumps(value)
2983+
2984+ self.cursor.execute('select data from kv where key=?', [key])
2985+ exists = self.cursor.fetchone()
2986+
2987+ # Skip mutations to the same value
2988+ if exists:
2989+ if exists[0] == serialized:
2990+ return value
2991+
2992+ if not exists:
2993+ self.cursor.execute(
2994+ 'insert into kv (key, data) values (?, ?)',
2995+ (key, serialized))
2996+ else:
2997+ self.cursor.execute('''
2998+ update kv
2999+ set data = ?
3000+ where key = ?''', [serialized, key])
3001+
3002+ # Save
3003+ if not self.revision:
3004+ return value
3005+
3006+ self.cursor.execute(
3007+ 'select 1 from kv_revisions where key=? and revision=?',
3008+ [key, self.revision])
3009+ exists = self.cursor.fetchone()
3010+
3011+ if not exists:
3012+ self.cursor.execute(
3013+ '''insert into kv_revisions (
3014+ revision, key, data) values (?, ?, ?)''',
3015+ (self.revision, key, serialized))
3016+ else:
3017+ self.cursor.execute(
3018+ '''
3019+ update kv_revisions
3020+ set data = ?
3021+ where key = ?
3022+ and revision = ?''',
3023+ [serialized, key, self.revision])
3024+
3025+ return value
3026+
3027+ def delta(self, mapping, prefix):
3028+ """
3029+ return a delta containing values that have changed.
3030+ """
3031+ previous = self.getrange(prefix, strip=True)
3032+ if not previous:
3033+ pk = set()
3034+ else:
3035+ pk = set(previous.keys())
3036+ ck = set(mapping.keys())
3037+ delta = DeltaSet()
3038+
3039+ # added
3040+ for k in ck.difference(pk):
3041+ delta[k] = Delta(None, mapping[k])
3042+
3043+ # removed
3044+ for k in pk.difference(ck):
3045+ delta[k] = Delta(previous[k], None)
3046+
3047+ # changed
3048+ for k in pk.intersection(ck):
3049+ c = mapping[k]
3050+ p = previous[k]
3051+ if c != p:
3052+ delta[k] = Delta(p, c)
3053+
3054+ return delta
3055+
3056+ @contextlib.contextmanager
3057+ def hook_scope(self, name=""):
3058+ """Scope all future interactions to the current hook execution
3059+ revision."""
3060+ assert not self.revision
3061+ self.cursor.execute(
3062+ 'insert into hooks (hook, date) values (?, ?)',
3063+ (name or sys.argv[0],
3064+ datetime.datetime.utcnow().isoformat()))
3065+ self.revision = self.cursor.lastrowid
3066+ try:
3067+ yield self.revision
3068+ self.revision = None
3069+ except:
3070+ self.flush(False)
3071+ self.revision = None
3072+ raise
3073+ else:
3074+ self.flush()
3075+
3076+ def flush(self, save=True):
3077+ if save:
3078+ self.conn.commit()
3079+ elif self._closed:
3080+ return
3081+ else:
3082+ self.conn.rollback()
3083+
3084+ def _init(self):
3085+ self.cursor.execute('''
3086+ create table if not exists kv (
3087+ key text,
3088+ data text,
3089+ primary key (key)
3090+ )''')
3091+ self.cursor.execute('''
3092+ create table if not exists kv_revisions (
3093+ key text,
3094+ revision integer,
3095+ data text,
3096+ primary key (key, revision)
3097+ )''')
3098+ self.cursor.execute('''
3099+ create table if not exists hooks (
3100+ version integer primary key autoincrement,
3101+ hook text,
3102+ date text
3103+ )''')
3104+ self.conn.commit()
3105+
3106+ def gethistory(self, key, deserialize=False):
3107+ self.cursor.execute(
3108+ '''
3109+ select kv.revision, kv.key, kv.data, h.hook, h.date
3110+ from kv_revisions kv,
3111+ hooks h
3112+ where kv.key=?
3113+ and kv.revision = h.version
3114+ ''', [key])
3115+ if deserialize is False:
3116+ return self.cursor.fetchall()
3117+ return map(_parse_history, self.cursor.fetchall())
3118+
3119+ def debug(self, fh=sys.stderr):
3120+ self.cursor.execute('select * from kv')
3121+ pprint.pprint(self.cursor.fetchall(), stream=fh)
3122+ self.cursor.execute('select * from kv_revisions')
3123+ pprint.pprint(self.cursor.fetchall(), stream=fh)
3124+
3125+
3126+def _parse_history(d):
3127+ return (d[0], d[1], json.loads(d[2]), d[3],
3128+ datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
3129+
3130+
3131+class HookData(object):
3132+ """Simple integration for existing hook exec frameworks.
3133+
3134+ Records all unit information, and stores deltas for processing
3135+ by the hook.
3136+
3137+ Sample::
3138+
3139+ from charmhelper.core import hookenv, unitdata
3140+
3141+ changes = unitdata.HookData()
3142+ db = unitdata.kv()
3143+ hooks = hookenv.Hooks()
3144+
3145+ @hooks.hook
3146+ def config_changed():
3147+ # View all changes to configuration
3148+ for changed, (prev, cur) in changes.conf.items():
3149+ print('config changed', changed,
3150+ 'previous value', prev,
3151+ 'current value', cur)
3152+
3153+ # Get some unit specific bookeeping
3154+ if not db.get('pkg_key'):
3155+ key = urllib.urlopen('https://example.com/pkg_key').read()
3156+ db.set('pkg_key', key)
3157+
3158+ if __name__ == '__main__':
3159+ with changes():
3160+ hook.execute()
3161+
3162+ """
3163+ def __init__(self):
3164+ self.kv = kv()
3165+ self.conf = None
3166+ self.rels = None
3167+
3168+ @contextlib.contextmanager
3169+ def __call__(self):
3170+ from charmhelpers.core import hookenv
3171+ hook_name = hookenv.hook_name()
3172+
3173+ with self.kv.hook_scope(hook_name):
3174+ self._record_charm_version(hookenv.charm_dir())
3175+ delta_config, delta_relation = self._record_hook(hookenv)
3176+ yield self.kv, delta_config, delta_relation
3177+
3178+ def _record_charm_version(self, charm_dir):
3179+ # Record revisions.. charm revisions are meaningless
3180+ # to charm authors as they don't control the revision.
3181+ # so logic dependnent on revision is not particularly
3182+ # useful, however it is useful for debugging analysis.
3183+ charm_rev = open(
3184+ os.path.join(charm_dir, 'revision')).read().strip()
3185+ charm_rev = charm_rev or '0'
3186+ revs = self.kv.get('charm_revisions', [])
3187+ if charm_rev not in revs:
3188+ revs.append(charm_rev.strip() or '0')
3189+ self.kv.set('charm_revisions', revs)
3190+
3191+ def _record_hook(self, hookenv):
3192+ data = hookenv.execution_environment()
3193+ self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
3194+ self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
3195+ self.kv.set('env', dict(data['env']))
3196+ self.kv.set('unit', data['unit'])
3197+ self.kv.set('relid', data.get('relid'))
3198+ return conf_delta, rels_delta
3199+
3200+
3201+class Record(dict):
3202+
3203+ __slots__ = ()
3204+
3205+ def __getattr__(self, k):
3206+ if k in self:
3207+ return self[k]
3208+ raise AttributeError(k)
3209+
3210+
3211+class DeltaSet(Record):
3212+
3213+ __slots__ = ()
3214+
3215+
3216+Delta = collections.namedtuple('Delta', ['previous', 'current'])
3217+
3218+
3219+_KV = None
3220+
3221+
3222+def kv():
3223+ global _KV
3224+ if _KV is None:
3225+ _KV = Storage()
3226+ return _KV
3227
3228=== modified file 'lib/charmhelpers/fetch/__init__.py'
3229--- lib/charmhelpers/fetch/__init__.py 2014-12-02 19:35:26 +0000
3230+++ lib/charmhelpers/fetch/__init__.py 2016-07-12 15:38:59 +0000
3231@@ -1,3 +1,19 @@
3232+# Copyright 2014-2015 Canonical Limited.
3233+#
3234+# This file is part of charm-helpers.
3235+#
3236+# charm-helpers is free software: you can redistribute it and/or modify
3237+# it under the terms of the GNU Lesser General Public License version 3 as
3238+# published by the Free Software Foundation.
3239+#
3240+# charm-helpers is distributed in the hope that it will be useful,
3241+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3242+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3243+# GNU Lesser General Public License for more details.
3244+#
3245+# You should have received a copy of the GNU Lesser General Public License
3246+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3247+
3248 import importlib
3249 from tempfile import NamedTemporaryFile
3250 import time
3251@@ -64,9 +80,40 @@
3252 'trusty-juno/updates': 'trusty-updates/juno',
3253 'trusty-updates/juno': 'trusty-updates/juno',
3254 'juno/proposed': 'trusty-proposed/juno',
3255- 'juno/proposed': 'trusty-proposed/juno',
3256 'trusty-juno/proposed': 'trusty-proposed/juno',
3257 'trusty-proposed/juno': 'trusty-proposed/juno',
3258+ # Kilo
3259+ 'kilo': 'trusty-updates/kilo',
3260+ 'trusty-kilo': 'trusty-updates/kilo',
3261+ 'trusty-kilo/updates': 'trusty-updates/kilo',
3262+ 'trusty-updates/kilo': 'trusty-updates/kilo',
3263+ 'kilo/proposed': 'trusty-proposed/kilo',
3264+ 'trusty-kilo/proposed': 'trusty-proposed/kilo',
3265+ 'trusty-proposed/kilo': 'trusty-proposed/kilo',
3266+ # Liberty
3267+ 'liberty': 'trusty-updates/liberty',
3268+ 'trusty-liberty': 'trusty-updates/liberty',
3269+ 'trusty-liberty/updates': 'trusty-updates/liberty',
3270+ 'trusty-updates/liberty': 'trusty-updates/liberty',
3271+ 'liberty/proposed': 'trusty-proposed/liberty',
3272+ 'trusty-liberty/proposed': 'trusty-proposed/liberty',
3273+ 'trusty-proposed/liberty': 'trusty-proposed/liberty',
3274+ # Mitaka
3275+ 'mitaka': 'trusty-updates/mitaka',
3276+ 'trusty-mitaka': 'trusty-updates/mitaka',
3277+ 'trusty-mitaka/updates': 'trusty-updates/mitaka',
3278+ 'trusty-updates/mitaka': 'trusty-updates/mitaka',
3279+ 'mitaka/proposed': 'trusty-proposed/mitaka',
3280+ 'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
3281+ 'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
3282+ # Newton
3283+ 'newton': 'xenial-updates/newton',
3284+ 'xenial-newton': 'xenial-updates/newton',
3285+ 'xenial-newton/updates': 'xenial-updates/newton',
3286+ 'xenial-updates/newton': 'xenial-updates/newton',
3287+ 'newton/proposed': 'xenial-proposed/newton',
3288+ 'xenial-newton/proposed': 'xenial-proposed/newton',
3289+ 'xenial-proposed/newton': 'xenial-proposed/newton',
3290 }
3291
3292 # The order of this list is very important. Handlers should be listed in from
3293@@ -135,7 +182,7 @@
3294
3295 def apt_cache(in_memory=True):
3296 """Build and return an apt cache"""
3297- import apt_pkg
3298+ from apt import apt_pkg
3299 apt_pkg.init()
3300 if in_memory:
3301 apt_pkg.config.set("Dir::Cache::pkgcache", "")
3302@@ -192,19 +239,27 @@
3303 _run_apt_command(cmd, fatal)
3304
3305
3306+def apt_mark(packages, mark, fatal=False):
3307+ """Flag one or more packages using apt-mark"""
3308+ log("Marking {} as {}".format(packages, mark))
3309+ cmd = ['apt-mark', mark]
3310+ if isinstance(packages, six.string_types):
3311+ cmd.append(packages)
3312+ else:
3313+ cmd.extend(packages)
3314+
3315+ if fatal:
3316+ subprocess.check_call(cmd, universal_newlines=True)
3317+ else:
3318+ subprocess.call(cmd, universal_newlines=True)
3319+
3320+
3321 def apt_hold(packages, fatal=False):
3322- """Hold one or more packages"""
3323- cmd = ['apt-mark', 'hold']
3324- if isinstance(packages, six.string_types):
3325- cmd.append(packages)
3326- else:
3327- cmd.extend(packages)
3328- log("Holding {}".format(packages))
3329-
3330- if fatal:
3331- subprocess.check_call(cmd)
3332- else:
3333- subprocess.call(cmd)
3334+ return apt_mark(packages, 'hold', fatal=fatal)
3335+
3336+
3337+def apt_unhold(packages, fatal=False):
3338+ return apt_mark(packages, 'unhold', fatal=fatal)
3339
3340
3341 def add_source(source, key=None):
3342@@ -343,15 +398,13 @@
3343 # We ONLY check for True here because can_handle may return a string
3344 # explaining why it can't handle a given source.
3345 handlers = [h for h in plugins() if h.can_handle(source) is True]
3346- installed_to = None
3347 for handler in handlers:
3348 try:
3349- installed_to = handler.install(source, *args, **kwargs)
3350- except UnhandledSource:
3351- pass
3352- if not installed_to:
3353- raise UnhandledSource("No handler found for source {}".format(source))
3354- return installed_to
3355+ return handler.install(source, *args, **kwargs)
3356+ except UnhandledSource as e:
3357+ log('Install source attempt unsuccessful: {}'.format(e),
3358+ level='WARNING')
3359+ raise UnhandledSource("No handler found for source {}".format(source))
3360
3361
3362 def install_from_config(config_var_name):
3363@@ -371,7 +424,7 @@
3364 importlib.import_module(package),
3365 classname)
3366 plugin_list.append(handler_class())
3367- except (ImportError, AttributeError):
3368+ except NotImplementedError:
3369 # Skip missing plugins so that they can be ommitted from
3370 # installation if desired
3371 log("FetchHandler {} not found, skipping plugin".format(
3372
3373=== modified file 'lib/charmhelpers/fetch/archiveurl.py'
3374--- lib/charmhelpers/fetch/archiveurl.py 2014-12-02 19:35:26 +0000
3375+++ lib/charmhelpers/fetch/archiveurl.py 2016-07-12 15:38:59 +0000
3376@@ -1,7 +1,33 @@
3377+# Copyright 2014-2015 Canonical Limited.
3378+#
3379+# This file is part of charm-helpers.
3380+#
3381+# charm-helpers is free software: you can redistribute it and/or modify
3382+# it under the terms of the GNU Lesser General Public License version 3 as
3383+# published by the Free Software Foundation.
3384+#
3385+# charm-helpers is distributed in the hope that it will be useful,
3386+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3387+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3388+# GNU Lesser General Public License for more details.
3389+#
3390+# You should have received a copy of the GNU Lesser General Public License
3391+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3392+
3393 import os
3394 import hashlib
3395 import re
3396
3397+from charmhelpers.fetch import (
3398+ BaseFetchHandler,
3399+ UnhandledSource
3400+)
3401+from charmhelpers.payload.archive import (
3402+ get_archive_handler,
3403+ extract,
3404+)
3405+from charmhelpers.core.host import mkdir, check_hash
3406+
3407 import six
3408 if six.PY3:
3409 from urllib.request import (
3410@@ -19,16 +45,6 @@
3411 )
3412 from urlparse import urlparse, urlunparse, parse_qs
3413
3414-from charmhelpers.fetch import (
3415- BaseFetchHandler,
3416- UnhandledSource
3417-)
3418-from charmhelpers.payload.archive import (
3419- get_archive_handler,
3420- extract,
3421-)
3422-from charmhelpers.core.host import mkdir, check_hash
3423-
3424
3425 def splituser(host):
3426 '''urllib.splituser(), but six's support of this seems broken'''
3427@@ -61,6 +77,8 @@
3428 def can_handle(self, source):
3429 url_parts = self.parse_url(source)
3430 if url_parts.scheme not in ('http', 'https', 'ftp', 'file'):
3431+ # XXX: Why is this returning a boolean and a string? It's
3432+ # doomed to fail since "bool(can_handle('foo://'))" will be True.
3433 return "Wrong source type"
3434 if get_archive_handler(self.base_url(source)):
3435 return True
3436@@ -90,7 +108,7 @@
3437 install_opener(opener)
3438 response = urlopen(source)
3439 try:
3440- with open(dest, 'w') as dest_file:
3441+ with open(dest, 'wb') as dest_file:
3442 dest_file.write(response.read())
3443 except Exception as e:
3444 if os.path.isfile(dest):
3445@@ -139,7 +157,11 @@
3446 else:
3447 algorithms = hashlib.algorithms_available
3448 if key in algorithms:
3449- check_hash(dld_file, value, key)
3450+ if len(value) != 1:
3451+ raise TypeError(
3452+ "Expected 1 hash value, not %d" % len(value))
3453+ expected = value[0]
3454+ check_hash(dld_file, expected, key)
3455 if checksum:
3456 check_hash(dld_file, checksum, hash_type)
3457 return extract(dld_file, dest)
3458
3459=== modified file 'lib/charmhelpers/fetch/bzrurl.py'
3460--- lib/charmhelpers/fetch/bzrurl.py 2014-12-02 19:35:26 +0000
3461+++ lib/charmhelpers/fetch/bzrurl.py 2016-07-12 15:38:59 +0000
3462@@ -1,54 +1,77 @@
3463+# Copyright 2014-2015 Canonical Limited.
3464+#
3465+# This file is part of charm-helpers.
3466+#
3467+# charm-helpers is free software: you can redistribute it and/or modify
3468+# it under the terms of the GNU Lesser General Public License version 3 as
3469+# published by the Free Software Foundation.
3470+#
3471+# charm-helpers is distributed in the hope that it will be useful,
3472+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3473+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3474+# GNU Lesser General Public License for more details.
3475+#
3476+# You should have received a copy of the GNU Lesser General Public License
3477+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3478+
3479 import os
3480+from subprocess import check_call
3481 from charmhelpers.fetch import (
3482 BaseFetchHandler,
3483- UnhandledSource
3484+ UnhandledSource,
3485+ filter_installed_packages,
3486+ apt_install,
3487 )
3488 from charmhelpers.core.host import mkdir
3489
3490-import six
3491-if six.PY3:
3492- raise ImportError('bzrlib does not support Python3')
3493
3494-try:
3495- from bzrlib.branch import Branch
3496-except ImportError:
3497- from charmhelpers.fetch import apt_install
3498- apt_install("python-bzrlib")
3499- from bzrlib.branch import Branch
3500+if filter_installed_packages(['bzr']) != []:
3501+ apt_install(['bzr'])
3502+ if filter_installed_packages(['bzr']) != []:
3503+ raise NotImplementedError('Unable to install bzr')
3504
3505
3506 class BzrUrlFetchHandler(BaseFetchHandler):
3507 """Handler for bazaar branches via generic and lp URLs"""
3508 def can_handle(self, source):
3509 url_parts = self.parse_url(source)
3510- if url_parts.scheme not in ('bzr+ssh', 'lp'):
3511+ if url_parts.scheme not in ('bzr+ssh', 'lp', ''):
3512 return False
3513+ elif not url_parts.scheme:
3514+ return os.path.exists(os.path.join(source, '.bzr'))
3515 else:
3516 return True
3517
3518- def branch(self, source, dest):
3519- url_parts = self.parse_url(source)
3520- # If we use lp:branchname scheme we need to load plugins
3521+ def branch(self, source, dest, revno=None):
3522 if not self.can_handle(source):
3523 raise UnhandledSource("Cannot handle {}".format(source))
3524- if url_parts.scheme == "lp":
3525- from bzrlib.plugin import load_plugins
3526- load_plugins()
3527- try:
3528- remote_branch = Branch.open(source)
3529- remote_branch.bzrdir.sprout(dest).open_branch()
3530- except Exception as e:
3531- raise e
3532+ cmd_opts = []
3533+ if revno:
3534+ cmd_opts += ['-r', str(revno)]
3535+ if os.path.exists(dest):
3536+ cmd = ['bzr', 'pull']
3537+ cmd += cmd_opts
3538+ cmd += ['--overwrite', '-d', dest, source]
3539+ else:
3540+ cmd = ['bzr', 'branch']
3541+ cmd += cmd_opts
3542+ cmd += [source, dest]
3543+ check_call(cmd)
3544
3545- def install(self, source):
3546+ def install(self, source, dest=None, revno=None):
3547 url_parts = self.parse_url(source)
3548 branch_name = url_parts.path.strip("/").split("/")[-1]
3549- dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
3550- branch_name)
3551- if not os.path.exists(dest_dir):
3552- mkdir(dest_dir, perms=0o755)
3553+ if dest:
3554+ dest_dir = os.path.join(dest, branch_name)
3555+ else:
3556+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
3557+ branch_name)
3558+
3559+ if dest and not os.path.exists(dest):
3560+ mkdir(dest, perms=0o755)
3561+
3562 try:
3563- self.branch(source, dest_dir)
3564+ self.branch(source, dest_dir, revno)
3565 except OSError as e:
3566 raise UnhandledSource(e.strerror)
3567 return dest_dir
3568
3569=== modified file 'lib/charmhelpers/fetch/giturl.py'
3570--- lib/charmhelpers/fetch/giturl.py 2014-12-02 19:35:26 +0000
3571+++ lib/charmhelpers/fetch/giturl.py 2016-07-12 15:38:59 +0000
3572@@ -1,20 +1,32 @@
3573+# Copyright 2014-2015 Canonical Limited.
3574+#
3575+# This file is part of charm-helpers.
3576+#
3577+# charm-helpers is free software: you can redistribute it and/or modify
3578+# it under the terms of the GNU Lesser General Public License version 3 as
3579+# published by the Free Software Foundation.
3580+#
3581+# charm-helpers is distributed in the hope that it will be useful,
3582+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3583+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3584+# GNU Lesser General Public License for more details.
3585+#
3586+# You should have received a copy of the GNU Lesser General Public License
3587+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3588+
3589 import os
3590+from subprocess import check_call, CalledProcessError
3591 from charmhelpers.fetch import (
3592 BaseFetchHandler,
3593- UnhandledSource
3594+ UnhandledSource,
3595+ filter_installed_packages,
3596+ apt_install,
3597 )
3598-from charmhelpers.core.host import mkdir
3599-
3600-import six
3601-if six.PY3:
3602- raise ImportError('GitPython does not support Python 3')
3603-
3604-try:
3605- from git import Repo
3606-except ImportError:
3607- from charmhelpers.fetch import apt_install
3608- apt_install("python-git")
3609- from git import Repo
3610+
3611+if filter_installed_packages(['git']) != []:
3612+ apt_install(['git'])
3613+ if filter_installed_packages(['git']) != []:
3614+ raise NotImplementedError('Unable to install git')
3615
3616
3617 class GitUrlFetchHandler(BaseFetchHandler):
3618@@ -22,27 +34,37 @@
3619 def can_handle(self, source):
3620 url_parts = self.parse_url(source)
3621 # TODO (mattyw) no support for ssh git@ yet
3622- if url_parts.scheme not in ('http', 'https', 'git'):
3623+ if url_parts.scheme not in ('http', 'https', 'git', ''):
3624 return False
3625+ elif not url_parts.scheme:
3626+ return os.path.exists(os.path.join(source, '.git'))
3627 else:
3628 return True
3629
3630- def clone(self, source, dest, branch):
3631+ def clone(self, source, dest, branch="master", depth=None):
3632 if not self.can_handle(source):
3633 raise UnhandledSource("Cannot handle {}".format(source))
3634
3635- repo = Repo.clone_from(source, dest)
3636- repo.git.checkout(branch)
3637+ if os.path.exists(dest):
3638+ cmd = ['git', '-C', dest, 'pull', source, branch]
3639+ else:
3640+ cmd = ['git', 'clone', source, dest, '--branch', branch]
3641+ if depth:
3642+ cmd.extend(['--depth', depth])
3643+ check_call(cmd)
3644
3645- def install(self, source, branch="master"):
3646+ def install(self, source, branch="master", dest=None, depth=None):
3647 url_parts = self.parse_url(source)
3648 branch_name = url_parts.path.strip("/").split("/")[-1]
3649- dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
3650- branch_name)
3651- if not os.path.exists(dest_dir):
3652- mkdir(dest_dir, perms=0o755)
3653+ if dest:
3654+ dest_dir = os.path.join(dest, branch_name)
3655+ else:
3656+ dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
3657+ branch_name)
3658 try:
3659- self.clone(source, dest_dir, branch)
3660+ self.clone(source, dest_dir, branch, depth)
3661+ except CalledProcessError as e:
3662+ raise UnhandledSource(e)
3663 except OSError as e:
3664 raise UnhandledSource(e.strerror)
3665 return dest_dir
3666
3667=== modified file 'lib/charmhelpers/payload/__init__.py'
3668--- lib/charmhelpers/payload/__init__.py 2014-12-02 19:35:26 +0000
3669+++ lib/charmhelpers/payload/__init__.py 2016-07-12 15:38:59 +0000
3670@@ -1,1 +1,17 @@
3671+# Copyright 2014-2015 Canonical Limited.
3672+#
3673+# This file is part of charm-helpers.
3674+#
3675+# charm-helpers is free software: you can redistribute it and/or modify
3676+# it under the terms of the GNU Lesser General Public License version 3 as
3677+# published by the Free Software Foundation.
3678+#
3679+# charm-helpers is distributed in the hope that it will be useful,
3680+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3681+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3682+# GNU Lesser General Public License for more details.
3683+#
3684+# You should have received a copy of the GNU Lesser General Public License
3685+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3686+
3687 "Tools for working with files injected into a charm just before deployment."
3688
3689=== modified file 'lib/charmhelpers/payload/archive.py'
3690--- lib/charmhelpers/payload/archive.py 2014-12-02 19:35:26 +0000
3691+++ lib/charmhelpers/payload/archive.py 2016-07-12 15:38:59 +0000
3692@@ -1,3 +1,19 @@
3693+# Copyright 2014-2015 Canonical Limited.
3694+#
3695+# This file is part of charm-helpers.
3696+#
3697+# charm-helpers is free software: you can redistribute it and/or modify
3698+# it under the terms of the GNU Lesser General Public License version 3 as
3699+# published by the Free Software Foundation.
3700+#
3701+# charm-helpers is distributed in the hope that it will be useful,
3702+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3703+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3704+# GNU Lesser General Public License for more details.
3705+#
3706+# You should have received a copy of the GNU Lesser General Public License
3707+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3708+
3709 import os
3710 import tarfile
3711 import zipfile
3712
3713=== modified file 'lib/charmhelpers/payload/execd.py'
3714--- lib/charmhelpers/payload/execd.py 2014-12-02 19:35:26 +0000
3715+++ lib/charmhelpers/payload/execd.py 2016-07-12 15:38:59 +0000
3716@@ -1,5 +1,21 @@
3717 #!/usr/bin/env python
3718
3719+# Copyright 2014-2015 Canonical Limited.
3720+#
3721+# This file is part of charm-helpers.
3722+#
3723+# charm-helpers is free software: you can redistribute it and/or modify
3724+# it under the terms of the GNU Lesser General Public License version 3 as
3725+# published by the Free Software Foundation.
3726+#
3727+# charm-helpers is distributed in the hope that it will be useful,
3728+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3729+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3730+# GNU Lesser General Public License for more details.
3731+#
3732+# You should have received a copy of the GNU Lesser General Public License
3733+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3734+
3735 import os
3736 import sys
3737 import subprocess
3738
3739=== modified file 'metadata.yaml'
3740--- metadata.yaml 2014-11-12 21:43:04 +0000
3741+++ metadata.yaml 2016-07-12 15:38:59 +0000
3742@@ -1,6 +1,7 @@
3743 name: mariadb
3744 summary: A Charm to install MariaDB
3745 maintainer: Daniel Bartholomew <dbart@mariadb.com>
3746+series: [trusty, xenial]
3747 description: |
3748 MariaDB is an open source database server. It can be used as the backing
3749 database for web, business, and other applications and application servers.
3750
3751=== modified file 'revision'
3752--- revision 2016-05-31 20:38:01 +0000
3753+++ revision 2016-07-12 15:38:59 +0000
3754@@ -1,1 +1,1 @@
3755-3
3756+2
3757
3758=== modified file 'scripts/charm_helpers_sync.py'
3759--- scripts/charm_helpers_sync.py 2014-12-02 19:35:26 +0000
3760+++ scripts/charm_helpers_sync.py 2016-07-12 15:38:59 +0000
3761@@ -1,5 +1,20 @@
3762-#!/usr/bin/env python
3763-# Copyright 2013 Canonical Ltd.
3764+#!/usr/bin/python
3765+
3766+# Copyright 2014-2015 Canonical Limited.
3767+#
3768+# This file is part of charm-helpers.
3769+#
3770+# charm-helpers is free software: you can redistribute it and/or modify
3771+# it under the terms of the GNU Lesser General Public License version 3 as
3772+# published by the Free Software Foundation.
3773+#
3774+# charm-helpers is distributed in the hope that it will be useful,
3775+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3776+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3777+# GNU Lesser General Public License for more details.
3778+#
3779+# You should have received a copy of the GNU Lesser General Public License
3780+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3781
3782 # Authors:
3783 # Adam Gandelman <adamg@ubuntu.com>
3784@@ -12,9 +27,10 @@
3785 import sys
3786 import tempfile
3787 import yaml
3788-
3789 from fnmatch import fnmatch
3790
3791+import six
3792+
3793 CHARM_HELPERS_BRANCH = 'lp:charm-helpers'
3794
3795
3796@@ -28,7 +44,7 @@
3797 def clone_helpers(work_dir, branch):
3798 dest = os.path.join(work_dir, 'charm-helpers')
3799 logging.info('Checking out %s to %s.' % (branch, dest))
3800- cmd = ['bzr', 'branch', branch, dest]
3801+ cmd = ['bzr', 'checkout', '--lightweight', branch, dest]
3802 subprocess.check_call(cmd)
3803 return dest
3804
3805@@ -119,6 +135,20 @@
3806
3807
3808 def sync(src, dest, module, opts=None):
3809+
3810+ # Sync charmhelpers/__init__.py for bootstrap code.
3811+ sync_pyfile(_src_path(src, '__init__'), dest)
3812+
3813+ # Sync other __init__.py files in the path leading to module.
3814+ m = []
3815+ steps = module.split('.')[:-1]
3816+ while steps:
3817+ m.append(steps.pop(0))
3818+ init = '.'.join(m + ['__init__'])
3819+ sync_pyfile(_src_path(src, init),
3820+ os.path.dirname(_dest_path(dest, init)))
3821+
3822+ # Sync the module, or maybe a .py file.
3823 if os.path.isdir(_src_path(src, module)):
3824 sync_directory(_src_path(src, module), _dest_path(dest, module), opts)
3825 elif _is_pyfile(_src_path(src, module)):
3826@@ -137,7 +167,7 @@
3827
3828 def extract_options(inc, global_options=None):
3829 global_options = global_options or []
3830- if global_options and isinstance(global_options, basestring):
3831+ if global_options and isinstance(global_options, six.string_types):
3832 global_options = [global_options]
3833 if '|' not in inc:
3834 return (inc, global_options)
3835@@ -147,7 +177,7 @@
3836
3837 def sync_helpers(include, src, dest, options=None):
3838 if not os.path.isdir(dest):
3839- os.mkdir(dest)
3840+ os.makedirs(dest)
3841
3842 global_options = parse_sync_options(options)
3843
3844@@ -157,7 +187,7 @@
3845 sync(src, dest, inc, opts)
3846 elif isinstance(inc, dict):
3847 # could also do nested dicts here.
3848- for k, v in inc.iteritems():
3849+ for k, v in six.iteritems(inc):
3850 if isinstance(v, list):
3851 for m in v:
3852 inc, opts = extract_options(m, global_options)
3853@@ -215,7 +245,7 @@
3854 checkout = clone_helpers(tmpd, config['branch'])
3855 sync_helpers(config['include'], checkout, config['destination'],
3856 options=sync_options)
3857- except Exception, e:
3858+ except Exception as e:
3859 logging.error("Could not sync: %s" % e)
3860 raise e
3861 finally:
3862
3863=== modified file 'tests/10-deploy-and-upgrade'
3864--- tests/10-deploy-and-upgrade 2015-11-03 22:05:55 +0000
3865+++ tests/10-deploy-and-upgrade 2016-07-12 15:38:59 +0000
3866@@ -11,17 +11,14 @@
3867 cls.deployment = amulet.Deployment(series='trusty')
3868
3869 mw_config = { 'name': 'MariaDB Test'}
3870- maria_config = { 'enterprise-eula': True, 'token': 'a202-wg2j'}
3871+
3872 cls.deployment.add('mariadb')
3873 cls.deployment.add('mediawiki')
3874 cls.deployment.configure('mediawiki', mw_config)
3875- cls.deployment.configure('mariadb', maria_config)
3876 cls.deployment.relate('mediawiki:db', 'mariadb:db')
3877 cls.deployment.expose('mediawiki')
3878
3879
3880-
3881-
3882 try:
3883 cls.deployment.setup(timeout=1200)
3884 cls.deployment.sentry.wait()
3885@@ -58,24 +55,7 @@
3886 response = requests.get(wiki_url)
3887 response.raise_for_status()
3888
3889-'''
3890- def test_enterprise_eval(self):
3891- self.deployment.configure('mariadb', {'enterprise-eula': True,
3892- 'token': 'a202-wg2j'})
3893-
3894- # Ensure the bintar was relocated
3895- dbunit = self.deployment.sentry.unit['mariadb/0']
3896-
3897- try:
3898- dbunit.directory_stat('/usr/local/mysql')
3899- amulet.raise_status(amulet.SKIP, 'bintar directory found, uncertain results ahead')
3900- except:
3901- # this is what we want to happen
3902- pass
3903-
3904- # re-run the test after in-place upgrade
3905- self.test_credentials()
3906-'''
3907+
3908
3909 if __name__ == '__main__':
3910 unittest.main()

Subscribers

People subscribed via source and target branches