Merge lp:~dbuliga/charms/trusty/nrpe/nrpe into lp:charms/trusty/nrpe

Proposed by Denis Buliga
Status: Needs review
Proposed branch: lp:~dbuliga/charms/trusty/nrpe/nrpe
Merge into: lp:charms/trusty/nrpe
Diff against target: 2433 lines (+1349/-658)
17 files modified
hooks/centos.py (+29/-0)
hooks/charmhelpers/__init__.py (+10/-0)
hooks/charmhelpers/core/host.py (+81/-288)
hooks/charmhelpers/core/host_factory/__init__.py (+492/-0)
hooks/charmhelpers/core/host_factory/centos/__init__.py (+53/-0)
hooks/charmhelpers/core/host_factory/ubuntu/__init__.py (+47/-0)
hooks/charmhelpers/fetch/__init__.py (+66/-285)
hooks/charmhelpers/fetch/bzrurl.py (+23/-32)
hooks/charmhelpers/fetch/centos/__init__.py (+158/-0)
hooks/charmhelpers/fetch/giturl.py (+24/-25)
hooks/charmhelpers/fetch/ubuntu/__init__.py (+296/-0)
hooks/nrpe_utils.py (+10/-12)
hooks/services.py (+7/-6)
hooks/ubuntu.py (+27/-0)
templates/nrpe-centos.tmpl (+16/-0)
tests/11-monitors-configurations (+5/-5)
tests/13-monitors-config (+5/-5)
To merge this branch: bzr merge lp:~dbuliga/charms/trusty/nrpe/nrpe
Reviewer Review Type Date Requested Status
Adam Israel (community) Needs Fixing
Review via email: mp+288615@code.launchpad.net

Description of the change

This branch introduces a factory that based on platform, loads the proper functionality for that platform. It deploys NRPE as a subordinate charm to a charm which is deployed on CentOS as well as for a charm which is deployed on Ubuntu.

A new parameter was added in the 'monitors' relation which is supposed to send the platform for the subordinate charm. This helps for instance, NAGIOS, to set icon and description for that charm based on the parameter.

The need for a template for CentOS appeared because that pid_file has to be under the user nrpe on CentOS instead of nagios on Ubuntu.

To post a comment you must log in.
lp:~dbuliga/charms/trusty/nrpe/nrpe updated
41. By David Ames

[gnuoy, r=thedac] Add support for multiple charms to be related to nrpe via the nrpe, juju-info or local-monitors relation. This is useful when joining an additional local subordinate to this charm. See README changes.

42. By Denis Buliga

Updated nrpe to work on CentOS

Revision history for this message
Adam Israel (aisrael) wrote :

Hi Denis,

I've had a chance to review your merge proposal. I've run into a few errors running the amulet tests related to missing config files. Could you check the logs at the pastebin link below and fix the tests? Once they pass, I'll be happy to re-review and get this promulgated.

http://pastebin.ubuntu.com/17298099/

review: Needs Fixing
Revision history for this message
Antonio Rosales (arosales) wrote :

@Denis,

Thanks for your contribution.

Given this is a CentoOS update this may actually be better as a separate CentOS charm. Specifically, the metadata stating it is a CentOS charm so Juju can deploy it as such and it can be discoverable in the Charm Store.

It looks like there is some feedback on the tests, but it would be great to see this in a stand-alone CentOS charm.

-thanks,
Antonio

Unmerged revisions

42. By Denis Buliga

Updated nrpe to work on CentOS

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== added file 'hooks/centos.py'
--- hooks/centos.py 1970-01-01 00:00:00 +0000
+++ hooks/centos.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,29 @@
1from charmhelpers.core import host
2from charmhelpers.core import hookenv
3from charmhelpers.core.services import helpers
4
5
6def determine_packages():
7 """ List of packages this charm needs installed """
8 pkgs = [
9 'epel-release',
10 'nagios',
11 'nagios-plugins-nrpe',
12 'nagios-plugins-all',
13 'nrpe'
14 ]
15 if hookenv.config('export_nagios_definitions'):
16 pkgs.append('rsync')
17 return pkgs
18
19
20def restart_nrpe(service_name):
21 """ Restart nrpe """
22 host.service_restart('nrpe')
23
24
25def render_nrpe_template():
26 return helpers.render_template(
27 source='nrpe-centos.tmpl',
28 target='/etc/nagios/nrpe.cfg'
29 )
030
=== modified file 'hooks/charmhelpers/__init__.py' (properties changed: -x to +x)
--- hooks/charmhelpers/__init__.py 2015-03-23 09:45:10 +0000
+++ hooks/charmhelpers/__init__.py 2016-04-29 11:54:41 +0000
@@ -18,6 +18,7 @@
18# only standard libraries.18# only standard libraries.
19import subprocess19import subprocess
20import sys20import sys
21import platform
2122
22try:23try:
23 import six # flake8: noqa24 import six # flake8: noqa
@@ -36,3 +37,12 @@
36 else:37 else:
37 subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])38 subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
38 import yaml # flake8: noqa39 import yaml # flake8: noqa
40
41
42def get_platform():
43 tuple_platform = platform.linux_distribution()
44 current_platform = tuple_platform[0]
45 if "Ubuntu" in current_platform:
46 return "ubuntu"
47 elif "CentOS" in current_platform:
48 return "centos"
3949
=== modified file 'hooks/charmhelpers/core/__init__.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/decorators.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/fstab.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/hookenv.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/host.py' (properties changed: -x to +x)
--- hooks/charmhelpers/core/host.py 2015-03-23 09:45:10 +0000
+++ hooks/charmhelpers/core/host.py 2016-04-29 11:54:41 +0000
@@ -21,236 +21,123 @@
21# Nick Moffitt <nick.moffitt@canonical.com>21# Nick Moffitt <nick.moffitt@canonical.com>
22# Matthew Wedgwood <matthew.wedgwood@canonical.com>22# Matthew Wedgwood <matthew.wedgwood@canonical.com>
2323
24import os
25import re
26import pwd
27import grp
28import random
29import string
30import subprocess
31import hashlib
32from contextlib import contextmanager24from contextlib import contextmanager
33from collections import OrderedDict25from charmhelpers.core.host_factory import Host
3426
35import six27host = Host()
36
37from .hookenv import log
38from .fstab import Fstab
3928
4029
41def service_start(service_name):30def service_start(service_name):
42 """Start a system service"""31 """Start a system service"""
43 return service('start', service_name)32 return host.service_start(service_name)
4433
4534
46def service_stop(service_name):35def service_stop(service_name):
47 """Stop a system service"""36 """Stop a system service"""
48 return service('stop', service_name)37 return host.service_stop(service_name)
4938
5039
51def service_restart(service_name):40def service_restart(service_name):
52 """Restart a system service"""41 """Restart a system service"""
53 return service('restart', service_name)42 return host.service_restart(service_name)
5443
5544
56def service_reload(service_name, restart_on_failure=False):45def service_reload(service_name, restart_on_failure=False):
57 """Reload a system service, optionally falling back to restart if46 """Reload a system service, optionally falling back to restart if
58 reload fails"""47 reload fails"""
59 service_result = service('reload', service_name)48 return host.service_reload(service_name, restart_on_failure)
60 if not service_result and restart_on_failure:
61 service_result = service('restart', service_name)
62 return service_result
6349
6450
65def service(action, service_name):51def service(action, service_name):
66 """Control a system service"""52 """Control a system service"""
67 cmd = ['service', service_name, action]53 return host.service(action, service_name)
68 return subprocess.call(cmd) == 0
6954
7055
71def service_running(service):56def service_running(service):
72 """Determine whether a system service is running"""57 return host.service_running(service)
73 try:
74 output = subprocess.check_output(
75 ['service', service, 'status'],
76 stderr=subprocess.STDOUT).decode('UTF-8')
77 except subprocess.CalledProcessError:
78 return False
79 else:
80 if ("start/running" in output or "is running" in output):
81 return True
82 else:
83 return False
8458
8559
86def service_available(service_name):60def service_available(service_name):
87 """Determine whether a system service is available"""61 """Determine whether a system service is available"""
88 try:62 return host.service_available(service_name)
89 subprocess.check_output(63
90 ['service', service_name, 'status'],64
91 stderr=subprocess.STDOUT).decode('UTF-8')65def adduser(username, password=None, shell='/bin/bash', system_user=False,
92 except subprocess.CalledProcessError as e:66 primary_group=None, secondary_groups=None):
93 return 'unrecognized service' not in e.output67 """
94 else:68 Add a user to the system.
95 return True69
9670 Will log but otherwise succeed if the user already exists.
9771
98def adduser(username, password=None, shell='/bin/bash', system_user=False):72 :param str username: Username to create
99 """Add a user to the system"""73 :param str password: Password for user; if ``None``, create a system user
100 try:74 :param str shell: The default shell for the user
101 user_info = pwd.getpwnam(username)75 :param bool system_user: Whether to create a login or system user
102 log('user {0} already exists!'.format(username))76 :param str primary_group: Primary group for user; defaults to their username
103 except KeyError:77 :param list secondary_groups: Optional list of additional groups
104 log('creating user {0}'.format(username))78
105 cmd = ['useradd']79 :returns: The password database entry struct, as returned by `pwd.getpwnam`
106 if system_user or password is None:80 """
107 cmd.append('--system')81 return host.adduser(username, password, shell, system_user,
108 else:82 primary_group, secondary_groups)
109 cmd.extend([83
110 '--create-home',84
111 '--shell', shell,85def user_exists(username):
112 '--password', password,86 """Check if a user exists"""
113 ])87 return host.user_exists(username)
114 cmd.append(username)
115 subprocess.check_call(cmd)
116 user_info = pwd.getpwnam(username)
117 return user_info
11888
11989
120def add_group(group_name, system_group=False):90def add_group(group_name, system_group=False):
121 """Add a group to the system"""91 return host.add_group(group_name, system_group)
122 try:
123 group_info = grp.getgrnam(group_name)
124 log('group {0} already exists!'.format(group_name))
125 except KeyError:
126 log('creating group {0}'.format(group_name))
127 cmd = ['addgroup']
128 if system_group:
129 cmd.append('--system')
130 else:
131 cmd.extend([
132 '--group',
133 ])
134 cmd.append(group_name)
135 subprocess.check_call(cmd)
136 group_info = grp.getgrnam(group_name)
137 return group_info
13892
13993
140def add_user_to_group(username, group):94def add_user_to_group(username, group):
141 """Add a user to a group"""95 host.add_user_to_group(username, group)
142 cmd = [
143 'gpasswd', '-a',
144 username,
145 group
146 ]
147 log("Adding user {} to group {}".format(username, group))
148 subprocess.check_call(cmd)
14996
15097
151def rsync(from_path, to_path, flags='-r', options=None):98def rsync(from_path, to_path, flags='-r', options=None):
152 """Replicate the contents of a path"""99 """Replicate the contents of a path"""
153 options = options or ['--delete', '--executability']100 return host.rsync(from_path, to_path, flags, options)
154 cmd = ['/usr/bin/rsync', flags]
155 cmd.extend(options)
156 cmd.append(from_path)
157 cmd.append(to_path)
158 log(" ".join(cmd))
159 return subprocess.check_output(cmd).decode('UTF-8').strip()
160101
161102
162def symlink(source, destination):103def symlink(source, destination):
163 """Create a symbolic link"""104 """Create a symbolic link"""
164 log("Symlinking {} as {}".format(source, destination))105 host.symlink(source, destination)
165 cmd = [
166 'ln',
167 '-sf',
168 source,
169 destination,
170 ]
171 subprocess.check_call(cmd)
172106
173107
174def mkdir(path, owner='root', group='root', perms=0o555, force=False):108def mkdir(path, owner='root', group='root', perms=0o555, force=False):
175 """Create a directory"""109 """Create a directory"""
176 log("Making dir {} {}:{} {:o}".format(path, owner, group,110 host.mkdir(path, owner, group, perms, force)
177 perms))
178 uid = pwd.getpwnam(owner).pw_uid
179 gid = grp.getgrnam(group).gr_gid
180 realpath = os.path.abspath(path)
181 path_exists = os.path.exists(realpath)
182 if path_exists and force:
183 if not os.path.isdir(realpath):
184 log("Removing non-directory file {} prior to mkdir()".format(path))
185 os.unlink(realpath)
186 os.makedirs(realpath, perms)
187 elif not path_exists:
188 os.makedirs(realpath, perms)
189 os.chown(realpath, uid, gid)
190 os.chmod(realpath, perms)
191111
192112
193def write_file(path, content, owner='root', group='root', perms=0o444):113def write_file(path, content, owner='root', group='root', perms=0o444):
194 """Create or overwrite a file with the contents of a byte string."""114 """Create or overwrite a file with the contents of a byte string."""
195 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))115 host.write_file(path, content, owner, group, perms)
196 uid = pwd.getpwnam(owner).pw_uid
197 gid = grp.getgrnam(group).gr_gid
198 with open(path, 'wb') as target:
199 os.fchown(target.fileno(), uid, gid)
200 os.fchmod(target.fileno(), perms)
201 target.write(content)
202116
203117
204def fstab_remove(mp):118def fstab_remove(mp):
205 """Remove the given mountpoint entry from /etc/fstab119 """Remove the given mountpoint entry from /etc/fstab"""
206 """120 return host.fstab_remove(mp)
207 return Fstab.remove_by_mountpoint(mp)
208121
209122
210def fstab_add(dev, mp, fs, options=None):123def fstab_add(dev, mp, fs, options=None):
211 """Adds the given device entry to the /etc/fstab file124 """Adds the given device entry to the /etc/fstab file"""
212 """125 return host.fstab_add(dev, mp, fs, options)
213 return Fstab.add(dev, mp, fs, options=options)
214126
215127
216def mount(device, mountpoint, options=None, persist=False, filesystem="ext3"):128def mount(device, mountpoint, options=None, persist=False, filesystem="ext3"):
217 """Mount a filesystem at a particular mountpoint"""129 """Mount a filesystem at a particular mountpoint"""
218 cmd_args = ['mount']130 return host.mount(device, mountpoint, options, persist, filesystem)
219 if options is not None:
220 cmd_args.extend(['-o', options])
221 cmd_args.extend([device, mountpoint])
222 try:
223 subprocess.check_output(cmd_args)
224 except subprocess.CalledProcessError as e:
225 log('Error mounting {} at {}\n{}'.format(device, mountpoint, e.output))
226 return False
227
228 if persist:
229 return fstab_add(device, mountpoint, filesystem, options=options)
230 return True
231131
232132
233def umount(mountpoint, persist=False):133def umount(mountpoint, persist=False):
234 """Unmount a filesystem"""134 """Unmount a filesystem"""
235 cmd_args = ['umount', mountpoint]135 return host.umount(mountpoint, persist)
236 try:
237 subprocess.check_output(cmd_args)
238 except subprocess.CalledProcessError as e:
239 log('Error unmounting {}\n{}'.format(mountpoint, e.output))
240 return False
241
242 if persist:
243 return fstab_remove(mountpoint)
244 return True
245136
246137
247def mounts():138def mounts():
248 """Get a list of all mounted volumes as [[mountpoint,device],[...]]"""139 """Get a list of all mounted volumes as [[mountpoint,device],[...]]"""
249 with open('/proc/mounts') as f:140 return host.mounts()
250 # [['/mount/point','/dev/path'],[...]]
251 system_mounts = [m[1::-1] for m in [l.strip().split()
252 for l in f.readlines()]]
253 return system_mounts
254141
255142
256def file_hash(path, hash_type='md5'):143def file_hash(path, hash_type='md5'):
@@ -260,13 +147,7 @@
260 :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,147 :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
261 such as md5, sha1, sha256, sha512, etc.148 such as md5, sha1, sha256, sha512, etc.
262 """149 """
263 if os.path.exists(path):150 return host.file_hash(path, hash_type)
264 h = getattr(hashlib, hash_type)()
265 with open(path, 'rb') as source:
266 h.update(source.read())
267 return h.hexdigest()
268 else:
269 return None
270151
271152
272def check_hash(path, checksum, hash_type='md5'):153def check_hash(path, checksum, hash_type='md5'):
@@ -280,9 +161,7 @@
280 :raises ChecksumError: If the file fails the checksum161 :raises ChecksumError: If the file fails the checksum
281162
282 """163 """
283 actual_checksum = file_hash(path, hash_type)164 host.check_hash(path, checksum, hash_type)
284 if checksum != actual_checksum:
285 raise ChecksumError("'%s' != '%s'" % (checksum, actual_checksum))
286165
287166
288class ChecksumError(ValueError):167class ChecksumError(ValueError):
@@ -296,154 +175,68 @@
296175
297 @restart_on_change({176 @restart_on_change({
298 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]177 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
178 '/etc/apache/sites-enabled/*': [ 'apache2' ]
299 })179 })
300 def ceph_client_changed():180 def config_changed():
301 pass # your code here181 pass # your code here
302182
303 In this example, the cinder-api and cinder-volume services183 In this example, the cinder-api and cinder-volume services
304 would be restarted if /etc/ceph/ceph.conf is changed by the184 would be restarted if /etc/ceph/ceph.conf is changed by the
305 ceph_client_changed function.185 ceph_client_changed function. The apache2 service would be
186 restarted if any file matching the pattern got changed, created
187 or removed. Standard wildcards are supported, see documentation
188 for the 'glob' module for more information.
306 """189 """
307 def wrap(f):190 return host.restart_on_change(restart_map, stopstart)
308 def wrapped_f(*args, **kwargs):
309 checksums = {}
310 for path in restart_map:
311 checksums[path] = file_hash(path)
312 f(*args, **kwargs)
313 restarts = []
314 for path in restart_map:
315 if checksums[path] != file_hash(path):
316 restarts += restart_map[path]
317 services_list = list(OrderedDict.fromkeys(restarts))
318 if not stopstart:
319 for service_name in services_list:
320 service('restart', service_name)
321 else:
322 for action in ['stop', 'start']:
323 for service_name in services_list:
324 service(action, service_name)
325 return wrapped_f
326 return wrap
327191
328192
329def lsb_release():193def lsb_release():
330 """Return /etc/lsb-release in a dict"""194 """Return /etc/os-release in a dict"""
331 d = {}195 return host.lsb_release()
332 with open('/etc/lsb-release', 'r') as lsb:
333 for l in lsb:
334 k, v = l.split('=')
335 d[k.strip()] = v.strip()
336 return d
337196
338197
339def pwgen(length=None):198def pwgen(length=None):
340 """Generate a random pasword."""199 """Generate a random pasword."""
341 if length is None:200 return host.pwgen(length)
342 # A random length is ok to use a weak PRNG201
343 length = random.choice(range(35, 45))202
344 alphanumeric_chars = [203def list_nics(nic_type=None):
345 l for l in (string.ascii_letters + string.digits)
346 if l not in 'l0QD1vAEIOUaeiou']
347 # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
348 # actual password
349 random_generator = random.SystemRandom()
350 random_chars = [
351 random_generator.choice(alphanumeric_chars) for _ in range(length)]
352 return(''.join(random_chars))
353
354
355def list_nics(nic_type):
356 '''Return a list of nics of given type(s)'''204 '''Return a list of nics of given type(s)'''
357 if isinstance(nic_type, six.string_types):205 return host.list_nics(nic_type)
358 int_types = [nic_type]
359 else:
360 int_types = nic_type
361 interfaces = []
362 for int_type in int_types:
363 cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
364 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
365 ip_output = (line for line in ip_output if line)
366 for line in ip_output:
367 if line.split()[1].startswith(int_type):
368 matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
369 if matched:
370 interface = matched.groups()[0]
371 else:
372 interface = line.split()[1].replace(":", "")
373 interfaces.append(interface)
374
375 return interfaces
376206
377207
378def set_nic_mtu(nic, mtu):208def set_nic_mtu(nic, mtu):
379 '''Set MTU on a network interface'''209 '''Set MTU on a network interface'''
380 cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]210 host.set_nic_mtu(nic, mtu)
381 subprocess.check_call(cmd)
382211
383212
384def get_nic_mtu(nic):213def get_nic_mtu(nic):
385 cmd = ['ip', 'addr', 'show', nic]214 '''Get MTU of a network interface'''
386 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')215 return host.get_nic_mtu(nic)
387 mtu = ""
388 for line in ip_output:
389 words = line.split()
390 if 'mtu' in words:
391 mtu = words[words.index("mtu") + 1]
392 return mtu
393216
394217
395def get_nic_hwaddr(nic):218def get_nic_hwaddr(nic):
396 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]219 return host.get_nic_hwaddr(nic)
397 ip_output = subprocess.check_output(cmd).decode('UTF-8')
398 hwaddr = ""
399 words = ip_output.split()
400 if 'link/ether' in words:
401 hwaddr = words[words.index('link/ether') + 1]
402 return hwaddr
403220
404221
405def cmp_pkgrevno(package, revno, pkgcache=None):222def cmp_pkgrevno(package, revno, pkgcache=None):
406 '''Compare supplied revno with the revno of the installed package223 return host.cmp_pkgrevno(package, revno, pkgcache)
407
408 * 1 => Installed revno is greater than supplied arg
409 * 0 => Installed revno is the same as supplied arg
410 * -1 => Installed revno is less than supplied arg
411
412 This function imports apt_cache function from charmhelpers.fetch if
413 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
414 you call this function, or pass an apt_pkg.Cache() instance.
415 '''
416 import apt_pkg
417 if not pkgcache:
418 from charmhelpers.fetch import apt_cache
419 pkgcache = apt_cache()
420 pkg = pkgcache[package]
421 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
422224
423225
424@contextmanager226@contextmanager
425def chdir(d):227def chdir(d):
426 cur = os.getcwd()228 host.chdir(d)
427 try:229
428 yield os.chdir(d)230
429 finally:231def chownr(path, owner, group, follow_links=True, chowntopdir=False):
430 os.chdir(cur)232 """
431233 Recursively change user and group ownership of files and directories
432234 in given path. Doesn't chown path itself by default, only its children.
433def chownr(path, owner, group, follow_links=True):235
434 uid = pwd.getpwnam(owner).pw_uid236 :param bool follow_links: Also Chown links if True
435 gid = grp.getgrnam(group).gr_gid237 :param bool chowntopdir: Also chown path itself if True
436 if follow_links:238 """
437 chown = os.chown239 host.chownr(path, owner, group, follow_links, chowntopdir)
438 else:
439 chown = os.lchown
440
441 for root, dirs, files in os.walk(path):
442 for name in dirs + files:
443 full = os.path.join(root, name)
444 broken_symlink = os.path.lexists(full) and not os.path.exists(full)
445 if not broken_symlink:
446 chown(full, uid, gid)
447240
448241
449def lchownr(path, owner, group):242def lchownr(path, owner, group):
450243
=== added directory 'hooks/charmhelpers/core/host_factory'
=== added file 'hooks/charmhelpers/core/host_factory/__init__.py'
--- hooks/charmhelpers/core/host_factory/__init__.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/host_factory/__init__.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,492 @@
1import os
2import re
3import pwd
4import glob
5import grp
6import random
7import string
8import subprocess
9import hashlib
10import importlib
11import six
12
13from contextlib import contextmanager
14from collections import OrderedDict
15from ..hookenv import log
16from ..fstab import Fstab
17from charmhelpers import get_platform
18
19SYSTEMD_SYSTEM = '/run/systemd/system'
20
21
22class HostBase(object):
23
24 def service_start(self, service_name):
25 """Start a system service"""
26 return self.service('start', service_name)
27
28 def service_stop(self, service_name):
29 """Stop a system service"""
30 return self.service('stop', service_name)
31
32 def service_restart(self, service_name):
33 """Restart a system service"""
34 return self.service('restart', service_name)
35
36 def service_reload(self, service_name, restart_on_failure=False):
37 """Reload a system service, optionally falling back to restart if
38 reload fails"""
39 service_result = self.service('reload', service_name)
40 if not service_result and restart_on_failure:
41 service_result = self.service('restart', service_name)
42 return service_result
43
44 def service(self, action, service_name):
45 """Control a system service"""
46 if self.init_is_systemd():
47 cmd = ['systemctl', action, service_name]
48 else:
49 cmd = ['service', service_name, action]
50 return subprocess.call(cmd) == 0
51
52 def service_running(self, service_name):
53 """Determine whether a system service is running"""
54 if self.init_is_systemd():
55 return self.service('is-active', service_name)
56 else:
57 try:
58 output = subprocess.check_output(
59 ['service', service_name, 'status'],
60 stderr=subprocess.STDOUT).decode('UTF-8')
61 except subprocess.CalledProcessError:
62 return False
63 else:
64 if ("start/running" in output or "is running" in output):
65 return True
66 else:
67 return False
68
69 def service_available(self, service_name):
70 """Determine whether a system service is available"""
71 if self.init_is_systemd():
72 return self.service('is-enabled', service_name)
73 try:
74 subprocess.check_output(
75 ['service', service_name, 'status'],
76 stderr=subprocess.STDOUT).decode('UTF-8')
77 except subprocess.CalledProcessError as e:
78 return b'unrecognized service' not in e.output
79 else:
80 return True
81
82 def init_is_systemd(self):
83 """Return True if the host system uses systemd, False otherwise."""
84 return os.path.isdir(SYSTEMD_SYSTEM)
85
86 def adduser(self, username, password=None, shell='/bin/bash',
87 system_user=False, primary_group=None, secondary_groups=None):
88 """Add a user to the system.
89
90 Will log but otherwise succeed if the user already exists.
91
92 :param str username: Username to create
93 :param str password: Password for user;
94 if ``None``, create a system user
95 :param str shell: The default shell for the user
96 :param bool system_user: Whether to create a login or system user
97 :param str primary_group: Primary group for user; defaults to username
98 :param list secondary_groups: Optional list of additional groups
99
100 :returns: The password database entry struct,
101 as returned by `pwd.getpwnam`
102 """
103 try:
104 user_info = pwd.getpwnam(username)
105 log('user {0} already exists!'.format(username))
106 except KeyError:
107 log('creating user {0}'.format(username))
108 cmd = ['useradd']
109 if system_user or password is None:
110 cmd.append('--system')
111 else:
112 cmd.extend([
113 '--create-home',
114 '--shell', shell,
115 '--password', password,
116 ])
117 if not primary_group:
118 try:
119 grp.getgrnam(username)
120 primary_group = username # avoid "group exists" error
121 except KeyError:
122 pass
123 if primary_group:
124 cmd.extend(['-g', primary_group])
125 if secondary_groups:
126 cmd.extend(['-G', ','.join(secondary_groups)])
127 cmd.append(username)
128 subprocess.check_call(cmd)
129 user_info = pwd.getpwnam(username)
130 return user_info
131
132 def _add_group(self, group_name, system_group=False):
133 raise NotImplementedError()
134
135 def add_group(self, group_name, system_group=False):
136 try:
137 group_info = grp.getgrnam(group_name)
138 log('group {0} already exists!'.format(group_name))
139 except KeyError:
140 log('creating group {0}'.format(group_name))
141 self._add_group(group_name, system_group)
142 group_info = grp.getgrnam(group_name)
143 return group_info
144
145 def add_user_to_group(self, username, group):
146 """Add a user to a group"""
147 cmd = ['gpasswd', '-a', username, group]
148 log("Adding user {} to group {}".format(username, group))
149 subprocess.check_call(cmd)
150
151 def rsync(self, from_path, to_path, flags='-r', options=None):
152 """Replicate the contents of a path"""
153 options = options or ['--delete', '--executability']
154 cmd = ['/usr/bin/rsync', flags]
155 cmd.extend(options)
156 cmd.append(from_path)
157 cmd.append(to_path)
158 log(" ".join(cmd))
159 return subprocess.check_output(cmd).decode('UTF-8').strip()
160
161 def symlink(self, source, destination):
162 """Create a symbolic link"""
163 log("Symlinking {} as {}".format(source, destination))
164 cmd = [
165 'ln',
166 '-sf',
167 source,
168 destination,
169 ]
170 subprocess.check_call(cmd)
171
172 def mkdir(self, path, owner='root', group='root',
173 perms=0o555, force=False):
174 """Create a directory"""
175 log("Making dir {} {}:{} {:o}".format(path, owner, group,
176 perms))
177 uid = pwd.getpwnam(owner).pw_uid
178 gid = grp.getgrnam(group).gr_gid
179 realpath = os.path.abspath(path)
180 path_exists = os.path.exists(realpath)
181 if path_exists and force:
182 if not os.path.isdir(realpath):
183 log("Removing non-directory file {}"
184 " prior to mkdir()".format(path))
185 os.unlink(realpath)
186 os.makedirs(realpath, perms)
187 elif not path_exists:
188 os.makedirs(realpath, perms)
189 os.chown(realpath, uid, gid)
190 os.chmod(realpath, perms)
191
192 def write_file(self, path, content, owner='root',
193 group='root', perms=0o444):
194 """Create or overwrite a file with the contents of a byte string."""
195 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
196 uid = pwd.getpwnam(owner).pw_uid
197 gid = grp.getgrnam(group).gr_gid
198 with open(path, 'wb') as target:
199 os.fchown(target.fileno(), uid, gid)
200 os.fchmod(target.fileno(), perms)
201 target.write(content)
202
203 def fstab_remove(self, mp):
204 """Remove the given mountpoint entry from /etc/fstab"""
205 return Fstab.remove_by_mountpoint(mp)
206
207 def fstab_add(self, dev, mp, fs, options=None):
208 """Adds the given device entry to the /etc/fstab file"""
209 return Fstab.add(dev, mp, fs, options=options)
210
211 def mount(self, device, mountpoint, options=None,
212 persist=False, filesystem="ext3"):
213 """Mount a filesystem at a particular mountpoint"""
214 cmd_args = ['mount']
215 if options is not None:
216 cmd_args.extend(['-o', options])
217 cmd_args.extend([device, mountpoint])
218 try:
219 subprocess.check_output(cmd_args)
220 except subprocess.CalledProcessError as e:
221 log('Error mounting {} at {}\n{}'.format(
222 device, mountpoint, e.output))
223 return False
224
225 if persist:
226 return self.fstab_add(device, mountpoint,
227 filesystem, options=options)
228 return True
229
230 def umount(self, mountpoint, persist=False):
231 """Unmount a filesystem"""
232 cmd_args = ['umount', mountpoint]
233 try:
234 subprocess.check_output(cmd_args)
235 except subprocess.CalledProcessError as e:
236 log('Error unmounting {}\n{}'.format(mountpoint, e.output))
237 return False
238
239 if persist:
240 return self.fstab_remove(mountpoint)
241 return True
242
243 def mounts(self):
244 """Get a list of all mounted volumes as [[mountpoint,device],[...]]"""
245 with open('/proc/mounts') as f:
246 # [['/mount/point','/dev/path'],[...]]
247 system_mounts = [m[1::-1] for m in [l.strip().split()
248 for l in f.readlines()]]
249 return system_mounts
250
251 def file_hash(self, path, hash_type='md5'):
252 """Generate a hash checksum of the contents of 'path' or None if not found.
253
254 :param str hash_type: Any hash alrgorithm supported by :mod:`hashlib`,
255 such as md5, sha1, sha256, sha512, etc.
256 """
257 if os.path.exists(path):
258 h = getattr(hashlib, hash_type)()
259 with open(path, 'rb') as source:
260 h.update(source.read())
261 return h.hexdigest()
262 else:
263 return None
264
265 def check_hash(self, path, checksum, hash_type='md5'):
266 """Validate a file using a cryptographic checksum.
267
268 :param str checksum: Value of the checksum used to validate the file.
269 :param str hash_type: Hash algorithm used to generate `checksum`.
270 Can be any hash alrgorithm supported by :mod:`hashlib`,
271 such as md5, sha1, sha256, sha512, etc.
272 :raises ChecksumError: If the file fails the checksum
273
274 """
275 actual_checksum = self.file_hash(path, hash_type)
276 if checksum != actual_checksum:
277 raise ChecksumError("'%s' != '%s'" % (checksum, actual_checksum))
278
279 def restart_on_change(self, restart_map, stopstart=False):
280 """Restart services based on configuration files changing
281
282 This function is used a decorator, for example::
283
284 @restart_on_change({
285 '/etc/ceph/ceph.conf': [ 'cinder-api', 'cinder-volume' ]
286 '/etc/apache/sites-enabled/*': [ 'apache2' ]
287 })
288 def config_changed():
289 pass # your code here
290
291 In this example, the cinder-api and cinder-volume services
292 would be restarted if /etc/ceph/ceph.conf is changed by the
293 ceph_client_changed function. The apache2 service would be
294 restarted if any file matching the pattern got changed, created
295 or removed. Standard wildcards are supported, see documentation
296 for the 'glob' module for more information.
297 """
298 def wrap(f):
299 def wrapped_f(*args, **kwargs):
300 checksums = {path: self.path_hash(path)
301 for path in restart_map}
302 f(*args, **kwargs)
303 restarts = []
304 for path in restart_map:
305 if self.path_hash(path) != checksums[path]:
306 restarts += restart_map[path]
307 services_list = list(OrderedDict.fromkeys(restarts))
308 if not stopstart:
309 for service_name in services_list:
310 self.service('restart', service_name)
311 else:
312 for action in ['stop', 'start']:
313 for service_name in services_list:
314 self.service(action, service_name)
315 return wrapped_f
316 return wrap
317
318 def lsb_release(self):
319 return self._lsb_release()
320
321 def _lsb_release(self):
322 raise NotImplementedError()
323
324 def pwgen(self, length=None):
325 """Generate a random pasword."""
326 if length is None:
327 # A random length is ok to use a weak PRNG
328 length = random.choice(range(35, 45))
329 alphanumeric_chars = [
330 l for l in (string.ascii_letters + string.digits)
331 if l not in 'l0QD1vAEIOUaeiou']
332 # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
333 # actual password
334 random_generator = random.SystemRandom()
335 random_chars = [
336 random_generator.choice(alphanumeric_chars) for _ in range(length)]
337 return(''.join(random_chars))
338
339 def list_nics(self, nic_type=None):
340 """Return a list of nics of given type(s)"""
341 if isinstance(nic_type, six.string_types):
342 int_types = [nic_type]
343 else:
344 int_types = nic_type
345
346 interfaces = []
347 if nic_type:
348 for int_type in int_types:
349 cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
350 ip_output = subprocess.check_output(cmd).decode('UTF-8')
351 ip_output = ip_output.split('\n')
352 ip_output = (line for line in ip_output if line)
353 for line in ip_output:
354 if line.split()[1].startswith(int_type):
355 matched = re.search('.*: (' + int_type +
356 r'[0-9]+\.[0-9]+)@.*', line)
357 if matched:
358 iface = matched.groups()[0]
359 else:
360 iface = line.split()[1].replace(":", "")
361
362 if iface not in interfaces:
363 interfaces.append(iface)
364 else:
365 cmd = ['ip', 'a']
366 ip_output = subprocess.check_output(
367 cmd).decode('UTF-8').split('\n')
368 ip_output = (line.strip() for line in ip_output if line)
369
370 key = re.compile('^[0-9]+:\s+(.+):')
371 for line in ip_output:
372 matched = re.search(key, line)
373 if matched:
374 iface = matched.group(1)
375 iface = iface.partition("@")[0]
376 if iface not in interfaces:
377 interfaces.append(iface)
378 return interfaces
379
380 def set_nic_mtu(self, nic, mtu):
381 """Set the Maximum Transmission Unit (MTU) on a network interface."""
382 cmd = ['ip', 'link', 'set', nic, 'mtu', mtu]
383 subprocess.check_call(cmd)
384
385 def get_nic_mtu(self, nic):
386 """
387 Return the Maximum Transmission Unit (MTU) for a network interface.
388 """
389 cmd = ['ip', 'addr', 'show', nic]
390 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
391 mtu = ""
392 for line in ip_output:
393 words = line.split()
394 if 'mtu' in words:
395 mtu = words[words.index("mtu") + 1]
396 return mtu
397
398 def get_nic_hwaddr(self, nic):
399 """Return the Media Access Control (MAC) for a network interface."""
400 cmd = ['ip', '-o', '-0', 'addr', 'show', nic]
401 ip_output = subprocess.check_output(cmd).decode('UTF-8')
402 hwaddr = ""
403 words = ip_output.split()
404 if 'link/ether' in words:
405 hwaddr = words[words.index('link/ether') + 1]
406 return hwaddr
407
408 def cmp_pkgrevno(self, package, revno, pkgcache=None):
409 """Compare supplied revno with the revno of the installed package
410
411 * 1 => Installed revno is greater than supplied arg
412 * 0 => Installed revno is the same as supplied arg
413 * -1 => Installed revno is less than supplied arg
414
415 This function imports apt_cache function from charmhelpers.fetch if
416 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
417 you call this function, or pass an apt_pkg.Cache() instance.
418 """
419 return self._cmp_pkgrevno(package, revno, pkgcache)
420
421 def _cmp_pkgrevno(self, package, revno, pkgcache=None):
422 raise NotImplementedError()
423
424 @contextmanager
425 def chdir(self, directory):
426 """
427 Change the current working directory to a different directory for a
428 code block and return the previous directory after the block exits.
429 Useful to run commands from a specificed directory.
430
431 :param str directory: The directory path to change to for this context.
432 """
433 cur = os.getcwd()
434 try:
435 yield os.chdir(directory)
436 finally:
437 os.chdir(cur)
438
439 def chownr(self, path, owner, group, follow_links=True, chowntopdir=False):
440 """Recursively change user and group ownership of files and directories
441 in given path. Doesn't chown path itself by default, only its children.
442
443 :param str path: The string path to start changing ownership.
444 :param str owner: The owner string to use when looking up the uid.
445 :param str group: The group string to use when looking up the gid.
446 :param bool follow_links: Also Chown links if True
447 :param bool chowntopdir: Also chown path itself if True
448 """
449 uid = pwd.getpwnam(owner).pw_uid
450 gid = grp.getgrnam(group).gr_gid
451 if follow_links:
452 chown = os.chown
453 else:
454 chown = os.lchown
455
456 if chowntopdir:
457 broken_symlink = os.path.lexists(path) and not os.path.exists(path)
458 if not broken_symlink:
459 chown(path, uid, gid)
460 for root, dirs, files in os.walk(path):
461 for name in dirs + files:
462 full = os.path.join(root, name)
463 broken_symlink = os.path.lexists(
464 full
465 ) and not os.path.exists(full)
466 if not broken_symlink:
467 chown(full, uid, gid)
468
469 def lchownr(self, path, owner, group):
470 """
471 Recursively change user and group ownership of files and directories
472 in a given path, not following symbolic links. See the documentation
473 for 'os.lchown' for more information.
474
475 :param str path: The string path to start changing ownership.
476 :param str owner: The owner string to use when looking up the uid.
477 :param str group: The group string to use when looking up the gid.
478 """
479 self.chownr(path, owner, group, follow_links=False)
480
481
482class ChecksumError(ValueError):
483 """A class derived from Value error to indicate the checksum failed."""
484 pass
485
486
487module = "charmhelpers.core.host_factory.%s" % get_platform()
488host = importlib.import_module(module)
489
490
491class Host(host.Host):
492 pass
0493
=== added directory 'hooks/charmhelpers/core/host_factory/centos'
=== added file 'hooks/charmhelpers/core/host_factory/centos/__init__.py'
--- hooks/charmhelpers/core/host_factory/centos/__init__.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/host_factory/centos/__init__.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,53 @@
1import subprocess
2import yum
3
4from .. import HostBase
5
6
7class Host(HostBase):
8 '''
9 Implementation of HostBase for CentOS
10 '''
11
12 def _add_group(self, group_name, system_group=False):
13 cmd = ['groupadd']
14 if system_group:
15 cmd.append('-r')
16 cmd.append(group_name)
17 subprocess.check_call(cmd)
18
19 def _lsb_release(self):
20 """Return /etc/os-release in a dict"""
21 d = {}
22 with open('/etc/os-release', 'r') as lsb:
23 for l in lsb:
24 if len(l.split('=')) != 2:
25 continue
26 k, v = l.split('=')
27 d[k.strip()] = v.strip()
28 return d
29
30 def _cmp_pkgrevno(self, package, revno, pkgcache=None):
31 """Compare supplied revno with the revno of the installed package
32
33 * 1 => Installed revno is greater than supplied arg
34 * 0 => Installed revno is the same as supplied arg
35 * -1 => Installed revno is less than supplied arg
36
37 This function imports apt_cache function from charmhelpers.fetch if
38 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
39 you call this function, or pass an apt_pkg.Cache() instance.
40 """
41 if not pkgcache:
42 y = yum.YumBase()
43 packages = y.doPackageLists()
44 pck = {}
45 for i in packages["installed"]:
46 pck[i.Name] = i.version
47 pkgcache = pck
48 pkg = pkgcache[package]
49 if pkg > revno:
50 return 1
51 if pkg < revno:
52 return -1
53 return 0
054
=== added directory 'hooks/charmhelpers/core/host_factory/ubuntu'
=== added file 'hooks/charmhelpers/core/host_factory/ubuntu/__init__.py'
--- hooks/charmhelpers/core/host_factory/ubuntu/__init__.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/host_factory/ubuntu/__init__.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,47 @@
1import subprocess
2
3from .. import HostBase
4
5
6class Host(HostBase):
7 '''
8 Implementation of HostBase for Ubuntu
9 '''
10
11 def _add_group(self, group_name, system_group=False):
12 cmd = ['addgroup']
13 if system_group:
14 cmd.append('--system')
15 else:
16 cmd.extend([
17 '--group',
18 ])
19 cmd.append(group_name)
20 subprocess.check_call(cmd)
21
22 def _lsb_release(self):
23 """Return /etc/lsb-release in a dict"""
24 d = {}
25 with open('/etc/lsb-release', 'r') as lsb:
26 for l in lsb:
27 k, v = l.split('=')
28 d[k.strip()] = v.strip()
29 return d
30
31 def _cmp_pkgrevno(self, package, revno, pkgcache=None):
32 """Compare supplied revno with the revno of the installed package
33
34 * 1 => Installed revno is greater than supplied arg
35 * 0 => Installed revno is the same as supplied arg
36 * -1 => Installed revno is less than supplied arg
37
38 This function imports apt_cache function from charmhelpers.fetch if
39 the pkgcache argument is None. Be sure to add charmhelpers.fetch if
40 you call this function, or pass an apt_pkg.Cache() instance.
41 """
42 import apt_pkg
43 if not pkgcache:
44 from charmhelpers.fetch import apt_cache
45 pkgcache = apt_cache()
46 pkg = pkgcache[package]
47 return apt_pkg.version_compare(pkg.current_ver.ver_str, revno)
048
=== modified file 'hooks/charmhelpers/core/services/__init__.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/services/base.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/services/helpers.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/strutils.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/sysctl.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/templating.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/core/unitdata.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/fetch/__init__.py' (properties changed: -x to +x)
--- hooks/charmhelpers/fetch/__init__.py 2015-03-23 09:45:10 +0000
+++ hooks/charmhelpers/fetch/__init__.py 2016-04-29 11:54:41 +0000
@@ -14,84 +14,21 @@
14# You should have received a copy of the GNU Lesser General Public License14# You should have received a copy of the GNU Lesser General Public License
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1616
17import six
17import importlib18import importlib
18from tempfile import NamedTemporaryFile19
19import time
20from yaml import safe_load20from yaml import safe_load
21from charmhelpers.core.host import (21from charmhelpers import get_platform
22 lsb_release22from charmhelpers.core.hookenv import(
23)
24import subprocess
25from charmhelpers.core.hookenv import (
26 config,23 config,
27 log,24 log
28)25)
29import os
30
31import six
32if six.PY3:26if six.PY3:
33 from urllib.parse import urlparse, urlunparse27 from urllib.parse import urlparse, urlunparse
34else:28else:
35 from urlparse import urlparse, urlunparse29 from urlparse import urlparse, urlunparse
3630
3731
38CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
39deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
40"""
41PROPOSED_POCKET = """# Proposed
42deb http://archive.ubuntu.com/ubuntu {}-proposed main universe multiverse restricted
43"""
44CLOUD_ARCHIVE_POCKETS = {
45 # Folsom
46 'folsom': 'precise-updates/folsom',
47 'precise-folsom': 'precise-updates/folsom',
48 'precise-folsom/updates': 'precise-updates/folsom',
49 'precise-updates/folsom': 'precise-updates/folsom',
50 'folsom/proposed': 'precise-proposed/folsom',
51 'precise-folsom/proposed': 'precise-proposed/folsom',
52 'precise-proposed/folsom': 'precise-proposed/folsom',
53 # Grizzly
54 'grizzly': 'precise-updates/grizzly',
55 'precise-grizzly': 'precise-updates/grizzly',
56 'precise-grizzly/updates': 'precise-updates/grizzly',
57 'precise-updates/grizzly': 'precise-updates/grizzly',
58 'grizzly/proposed': 'precise-proposed/grizzly',
59 'precise-grizzly/proposed': 'precise-proposed/grizzly',
60 'precise-proposed/grizzly': 'precise-proposed/grizzly',
61 # Havana
62 'havana': 'precise-updates/havana',
63 'precise-havana': 'precise-updates/havana',
64 'precise-havana/updates': 'precise-updates/havana',
65 'precise-updates/havana': 'precise-updates/havana',
66 'havana/proposed': 'precise-proposed/havana',
67 'precise-havana/proposed': 'precise-proposed/havana',
68 'precise-proposed/havana': 'precise-proposed/havana',
69 # Icehouse
70 'icehouse': 'precise-updates/icehouse',
71 'precise-icehouse': 'precise-updates/icehouse',
72 'precise-icehouse/updates': 'precise-updates/icehouse',
73 'precise-updates/icehouse': 'precise-updates/icehouse',
74 'icehouse/proposed': 'precise-proposed/icehouse',
75 'precise-icehouse/proposed': 'precise-proposed/icehouse',
76 'precise-proposed/icehouse': 'precise-proposed/icehouse',
77 # Juno
78 'juno': 'trusty-updates/juno',
79 'trusty-juno': 'trusty-updates/juno',
80 'trusty-juno/updates': 'trusty-updates/juno',
81 'trusty-updates/juno': 'trusty-updates/juno',
82 'juno/proposed': 'trusty-proposed/juno',
83 'trusty-juno/proposed': 'trusty-proposed/juno',
84 'trusty-proposed/juno': 'trusty-proposed/juno',
85 # Kilo
86 'kilo': 'trusty-updates/kilo',
87 'trusty-kilo': 'trusty-updates/kilo',
88 'trusty-kilo/updates': 'trusty-updates/kilo',
89 'trusty-updates/kilo': 'trusty-updates/kilo',
90 'kilo/proposed': 'trusty-proposed/kilo',
91 'trusty-kilo/proposed': 'trusty-proposed/kilo',
92 'trusty-proposed/kilo': 'trusty-proposed/kilo',
93}
94
95# The order of this list is very important. Handlers should be listed in from32# The order of this list is very important. Handlers should be listed in from
96# least- to most-specific URL matching.33# least- to most-specific URL matching.
97FETCH_HANDLERS = (34FETCH_HANDLERS = (
@@ -100,10 +37,6 @@
100 'charmhelpers.fetch.giturl.GitUrlFetchHandler',37 'charmhelpers.fetch.giturl.GitUrlFetchHandler',
101)38)
10239
103APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
104APT_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
105APT_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
106
10740
108class SourceConfigError(Exception):41class SourceConfigError(Exception):
109 pass42 pass
@@ -141,162 +74,38 @@
141 return urlunparse(parts)74 return urlunparse(parts)
14275
14376
77module = "charmhelpers.fetch.%s" % get_platform()
78fetch = importlib.import_module(module)
79
80
144def filter_installed_packages(packages):81def filter_installed_packages(packages):
145 """Returns a list of packages that require installation"""82 """Returns a list of packages that require installation"""
146 cache = apt_cache()83 return fetch.filter_installed_packages(packages)
147 _pkgs = []84
148 for package in packages:85
149 try:86def install(packages, options=None, fatal=False):
150 p = cache[package]
151 p.current_ver or _pkgs.append(package)
152 except KeyError:
153 log('Package {} has no installation candidate.'.format(package),
154 level='WARNING')
155 _pkgs.append(package)
156 return _pkgs
157
158
159def apt_cache(in_memory=True):
160 """Build and return an apt cache"""
161 import apt_pkg
162 apt_pkg.init()
163 if in_memory:
164 apt_pkg.config.set("Dir::Cache::pkgcache", "")
165 apt_pkg.config.set("Dir::Cache::srcpkgcache", "")
166 return apt_pkg.Cache()
167
168
169def apt_install(packages, options=None, fatal=False):
170 """Install one or more packages"""87 """Install one or more packages"""
171 if options is None:88 fetch.install(packages, options, fatal)
172 options = ['--option=Dpkg::Options::=--force-confold']89
17390
174 cmd = ['apt-get', '--assume-yes']91def upgrade(options=None, fatal=False, dist=False):
175 cmd.extend(options)
176 cmd.append('install')
177 if isinstance(packages, six.string_types):
178 cmd.append(packages)
179 else:
180 cmd.extend(packages)
181 log("Installing {} with options: {}".format(packages,
182 options))
183 _run_apt_command(cmd, fatal)
184
185
186def apt_upgrade(options=None, fatal=False, dist=False):
187 """Upgrade all packages"""92 """Upgrade all packages"""
188 if options is None:93 fetch.upgrade(options, fatal, dist)
189 options = ['--option=Dpkg::Options::=--force-confold']94
19095
191 cmd = ['apt-get', '--assume-yes']96def update(fatal=False):
192 cmd.extend(options)
193 if dist:
194 cmd.append('dist-upgrade')
195 else:
196 cmd.append('upgrade')
197 log("Upgrading with options: {}".format(options))
198 _run_apt_command(cmd, fatal)
199
200
201def apt_update(fatal=False):
202 """Update local apt cache"""97 """Update local apt cache"""
203 cmd = ['apt-get', 'update']98 fetch.update(fatal)
204 _run_apt_command(cmd, fatal)99
205100
206101def purge(packages, fatal=False):
207def apt_purge(packages, fatal=False):
208 """Purge one or more packages"""102 """Purge one or more packages"""
209 cmd = ['apt-get', '--assume-yes', 'purge']103 fetch.purge(packages, fatal)
210 if isinstance(packages, six.string_types):104
211 cmd.append(packages)105
212 else:106# PPA only works with .DEB packed and not with .RPM
213 cmd.extend(packages)
214 log("Purging {}".format(packages))
215 _run_apt_command(cmd, fatal)
216
217
218def apt_hold(packages, fatal=False):
219 """Hold one or more packages"""
220 cmd = ['apt-mark', 'hold']
221 if isinstance(packages, six.string_types):
222 cmd.append(packages)
223 else:
224 cmd.extend(packages)
225 log("Holding {}".format(packages))
226
227 if fatal:
228 subprocess.check_call(cmd)
229 else:
230 subprocess.call(cmd)
231
232
233def add_source(source, key=None):107def add_source(source, key=None):
234 """Add a package source to this system.108 fetch.add_source(source, key)
235
236 @param source: a URL or sources.list entry, as supported by
237 add-apt-repository(1). Examples::
238
239 ppa:charmers/example
240 deb https://stub:key@private.example.com/ubuntu trusty main
241
242 In addition:
243 'proposed:' may be used to enable the standard 'proposed'
244 pocket for the release.
245 'cloud:' may be used to activate official cloud archive pockets,
246 such as 'cloud:icehouse'
247 'distro' may be used as a noop
248
249 @param key: A key to be added to the system's APT keyring and used
250 to verify the signatures on packages. Ideally, this should be an
251 ASCII format GPG public key including the block headers. A GPG key
252 id may also be used, but be aware that only insecure protocols are
253 available to retrieve the actual public key from a public keyserver
254 placing your Juju environment at risk. ppa and cloud archive keys
255 are securely added automtically, so sould not be provided.
256 """
257 if source is None:
258 log('Source is not present. Skipping')
259 return
260
261 if (source.startswith('ppa:') or
262 source.startswith('http') or
263 source.startswith('deb ') or
264 source.startswith('cloud-archive:')):
265 subprocess.check_call(['add-apt-repository', '--yes', source])
266 elif source.startswith('cloud:'):
267 apt_install(filter_installed_packages(['ubuntu-cloud-keyring']),
268 fatal=True)
269 pocket = source.split(':')[-1]
270 if pocket not in CLOUD_ARCHIVE_POCKETS:
271 raise SourceConfigError(
272 'Unsupported cloud: source option %s' %
273 pocket)
274 actual_pocket = CLOUD_ARCHIVE_POCKETS[pocket]
275 with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
276 apt.write(CLOUD_ARCHIVE.format(actual_pocket))
277 elif source == 'proposed':
278 release = lsb_release()['DISTRIB_CODENAME']
279 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
280 apt.write(PROPOSED_POCKET.format(release))
281 elif source == 'distro':
282 pass
283 else:
284 log("Unknown source: {!r}".format(source))
285
286 if key:
287 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
288 with NamedTemporaryFile('w+') as key_file:
289 key_file.write(key)
290 key_file.flush()
291 key_file.seek(0)
292 subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
293 else:
294 # Note that hkp: is in no way a secure protocol. Using a
295 # GPG key id is pointless from a security POV unless you
296 # absolutely trust your network and DNS.
297 subprocess.check_call(['apt-key', 'adv', '--keyserver',
298 'hkp://keyserver.ubuntu.com:80', '--recv',
299 key])
300109
301110
302def configure_sources(update=False,111def configure_sources(update=False,
@@ -338,7 +147,32 @@
338 for source, key in zip(sources, keys):147 for source, key in zip(sources, keys):
339 add_source(source, key)148 add_source(source, key)
340 if update:149 if update:
341 apt_update(fatal=True)150 update(fatal=True)
151
152
153def install_from_config(config_var_name):
154 charm_config = config()
155 source = charm_config[config_var_name]
156 return install_remote(source)
157
158
159def plugins(fetch_handlers=None):
160 if not fetch_handlers:
161 fetch_handlers = FETCH_HANDLERS
162 plugin_list = []
163 for handler_name in fetch_handlers:
164 package, classname = handler_name.rsplit('.', 1)
165 try:
166 handler_class = getattr(
167 importlib.import_module(package),
168 classname)
169 plugin_list.append(handler_class())
170 except NotImplementedError:
171 # Skip missing plugins so that they can be ommitted from
172 # installation if desired
173 log("FetchHandler {} not found, skipping plugin".format(
174 handler_name))
175 return plugin_list
342176
343177
344def install_remote(source, *args, **kwargs):178def install_remote(source, *args, **kwargs):
@@ -370,70 +204,17 @@
370 for handler in handlers:204 for handler in handlers:
371 try:205 try:
372 installed_to = handler.install(source, *args, **kwargs)206 installed_to = handler.install(source, *args, **kwargs)
373 except UnhandledSource:207 except UnhandledSource as e:
374 pass208 log('Install source attempt unsuccessful: {}'.format(e),
209 level='WARNING')
375 if not installed_to:210 if not installed_to:
376 raise UnhandledSource("No handler found for source {}".format(source))211 raise UnhandledSource("No handler found for source {}".format(source))
377 return installed_to212 return installed_to
378213
379214# Backwards compatibility
380def install_from_config(config_var_name):215if get_platform() == "ubuntu":
381 charm_config = config()216 from charmhelpers.fetch.ubuntu import *
382 source = charm_config[config_var_name]217 apt_install = install
383 return install_remote(source)218 apt_update = update
384219 apt_upgrade = upgrade
385220 apt_purge = purge
386def plugins(fetch_handlers=None):
387 if not fetch_handlers:
388 fetch_handlers = FETCH_HANDLERS
389 plugin_list = []
390 for handler_name in fetch_handlers:
391 package, classname = handler_name.rsplit('.', 1)
392 try:
393 handler_class = getattr(
394 importlib.import_module(package),
395 classname)
396 plugin_list.append(handler_class())
397 except (ImportError, AttributeError):
398 # Skip missing plugins so that they can be ommitted from
399 # installation if desired
400 log("FetchHandler {} not found, skipping plugin".format(
401 handler_name))
402 return plugin_list
403
404
405def _run_apt_command(cmd, fatal=False):
406 """
407 Run an APT command, checking output and retrying if the fatal flag is set
408 to True.
409
410 :param: cmd: str: The apt command to run.
411 :param: fatal: bool: Whether the command's output should be checked and
412 retried.
413 """
414 env = os.environ.copy()
415
416 if 'DEBIAN_FRONTEND' not in env:
417 env['DEBIAN_FRONTEND'] = 'noninteractive'
418
419 if fatal:
420 retry_count = 0
421 result = None
422
423 # If the command is considered "fatal", we need to retry if the apt
424 # lock was not acquired.
425
426 while result is None or result == APT_NO_LOCK:
427 try:
428 result = subprocess.check_call(cmd, env=env)
429 except subprocess.CalledProcessError as e:
430 retry_count = retry_count + 1
431 if retry_count > APT_NO_LOCK_RETRY_COUNT:
432 raise
433 result = e.returncode
434 log("Couldn't acquire DPKG lock. Will retry in {} seconds."
435 "".format(APT_NO_LOCK_RETRY_DELAY))
436 time.sleep(APT_NO_LOCK_RETRY_DELAY)
437
438 else:
439 subprocess.call(cmd, env=env)
440221
=== modified file 'hooks/charmhelpers/fetch/archiveurl.py' (properties changed: -x to +x)
=== modified file 'hooks/charmhelpers/fetch/bzrurl.py' (properties changed: -x to +x)
--- hooks/charmhelpers/fetch/bzrurl.py 2015-03-23 09:45:10 +0000
+++ hooks/charmhelpers/fetch/bzrurl.py 2016-04-29 11:54:41 +0000
@@ -15,60 +15,51 @@
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1616
17import os17import os
18
19from subprocess import check_call
18from charmhelpers.fetch import (20from charmhelpers.fetch import (
19 BaseFetchHandler,21 BaseFetchHandler,
20 UnhandledSource22 UnhandledSource,
23 filter_installed_packages,
24 install,
21)25)
22from charmhelpers.core.host import mkdir26from charmhelpers.core.host import mkdir
2327
24import six
25if six.PY3:
26 raise ImportError('bzrlib does not support Python3')
2728
28try:29if filter_installed_packages(['bzr']) != []:
29 from bzrlib.branch import Branch30 install(['bzr'])
30 from bzrlib import bzrdir, workingtree, errors31 if filter_installed_packages(['bzr']) != []:
31except ImportError:32 raise NotImplementedError('Unable to install bzr')
32 from charmhelpers.fetch import apt_install
33 apt_install("python-bzrlib")
34 from bzrlib.branch import Branch
35 from bzrlib import bzrdir, workingtree, errors
3633
3734
38class BzrUrlFetchHandler(BaseFetchHandler):35class BzrUrlFetchHandler(BaseFetchHandler):
39 """Handler for bazaar branches via generic and lp URLs"""36 """Handler for bazaar branches via generic and lp URLs"""
40 def can_handle(self, source):37 def can_handle(self, source):
41 url_parts = self.parse_url(source)38 url_parts = self.parse_url(source)
42 if url_parts.scheme not in ('bzr+ssh', 'lp'):39 if url_parts.scheme not in ('bzr+ssh', 'lp', ''):
43 return False40 return False
41 elif not url_parts.scheme:
42 return os.path.exists(os.path.join(source, '.bzr'))
44 else:43 else:
45 return True44 return True
4645
47 def branch(self, source, dest):46 def branch(self, source, dest):
48 url_parts = self.parse_url(source)
49 # If we use lp:branchname scheme we need to load plugins
50 if not self.can_handle(source):47 if not self.can_handle(source):
51 raise UnhandledSource("Cannot handle {}".format(source))48 raise UnhandledSource("Cannot handle {}".format(source))
52 if url_parts.scheme == "lp":49 if os.path.exists(dest):
53 from bzrlib.plugin import load_plugins50 check_call(['bzr', 'pull', '--overwrite', '-d', dest, source])
54 load_plugins()51 else:
55 try:52 check_call(['bzr', 'branch', source, dest])
56 local_branch = bzrdir.BzrDir.create_branch_convenience(dest)
57 except errors.AlreadyControlDirError:
58 local_branch = Branch.open(dest)
59 try:
60 remote_branch = Branch.open(source)
61 remote_branch.push(local_branch)
62 tree = workingtree.WorkingTree.open(dest)
63 tree.update()
64 except Exception as e:
65 raise e
6653
67 def install(self, source):54 def install(self, source, dest=None):
68 url_parts = self.parse_url(source)55 url_parts = self.parse_url(source)
69 branch_name = url_parts.path.strip("/").split("/")[-1]56 branch_name = url_parts.path.strip("/").split("/")[-1]
70 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",57 if dest:
71 branch_name)58 dest_dir = os.path.join(dest, branch_name)
59 else:
60 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
61 branch_name)
62
72 if not os.path.exists(dest_dir):63 if not os.path.exists(dest_dir):
73 mkdir(dest_dir, perms=0o755)64 mkdir(dest_dir, perms=0o755)
74 try:65 try:
7566
=== added directory 'hooks/charmhelpers/fetch/centos'
=== added file 'hooks/charmhelpers/fetch/centos/__init__.py'
--- hooks/charmhelpers/fetch/centos/__init__.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/fetch/centos/__init__.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,158 @@
1import subprocess
2import os
3import time
4import six
5import yum
6
7from tempfile import NamedTemporaryFile
8from charmhelpers.core.hookenv import (
9 log,
10)
11
12YUM_NO_LOCK = 1 # The return code for "couldn't acquire lock" in YUM.
13YUM_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
14YUM_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
15
16
17def filter_installed_packages(packages):
18 """Returns a list of packages that require installation"""
19 yb = yum.YumBase()
20 pkgs = []
21 for package in yb.doPackageLists().installed:
22 pkgs.append(package.base_package_name)
23 _pkgs = []
24 for package in packages:
25 if package not in pkgs:
26 _pkgs.append(package)
27 return _pkgs
28
29
30def install(packages, options=None, fatal=False):
31 """Install one or more packages"""
32 cmd = ['yum', '--assumeyes']
33 if options is not None:
34 cmd.extend(options)
35 cmd.append('install')
36 if isinstance(packages, six.string_types):
37 cmd.append(packages)
38 else:
39 cmd.extend(packages)
40 log("Installing {} with options: {}".format(packages,
41 options))
42 _run_yum_command(cmd, fatal)
43
44
45def update(fatal=False):
46 """Update local yum cache"""
47 cmd = ['yum', 'update', '--assumeyes']
48 log("Update with fatal: {}".format(fatal))
49 _run_yum_command(cmd, fatal)
50
51
52def upgrade(options=None, fatal=False, dist=False):
53 """Upgrade all packages"""
54 cmd = ['yum', '--assumeyes']
55 if options is not None:
56 cmd.extend(options)
57 cmd.append('upgrade')
58 log("Upgrading with options: {}".format(options))
59 _run_yum_command(cmd, fatal)
60
61
62def purge(packages, fatal=False):
63 """Purge one or more packages"""
64 cmd = ['yum', 'remove', '--assumeyes']
65 if isinstance(packages, six.string_types):
66 cmd.append(packages)
67 else:
68 cmd.extend(packages)
69 log("Purging {}".format(packages))
70 _run_yum_command(cmd, fatal)
71
72
73def yum_search(packages):
74 """Search for a package"""
75 output = {}
76 cmd = ['yum', 'search']
77 if isinstance(packages, six.string_types):
78 cmd.append(packages)
79 else:
80 cmd.extend(packages)
81 log("Searching for {}".format(packages))
82 result = subprocess.check_output(cmd)
83 for package in list(packages):
84 if package not in result:
85 output[package] = False
86 else:
87 output[package] = True
88 return output
89
90
91def add_source(source, key=None):
92 if source is None:
93 log('Source is not present. Skipping')
94 return
95
96 if source.startswith('http'):
97 log("Add source: {!r}".format(source))
98
99 found = False
100 # search if already exists
101 directory = '/etc/yum.repos.d/'
102 for filename in os.listdir(directory):
103 with open(directory+filename, 'r') as rpm_file:
104 if source in rpm_file:
105 found = True
106
107 if not found:
108 # write in the charms.repo
109 with open(directory+'Charms.repo', 'a') as rpm_file:
110 rpm_file.write('[%s]\n' % source[7:].replace('/', '_'))
111 rpm_file.write('name=%s\n' % source[7:])
112 rpm_file.write('baseurl=%s\n\n' % source)
113 else:
114 log("Unknown source: {!r}".format(source))
115
116 if key:
117 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
118 with NamedTemporaryFile('w+') as key_file:
119 key_file.write(key)
120 key_file.flush()
121 key_file.seek(0)
122 subprocess.check_call(['rpm', '--import', key_file])
123 else:
124 subprocess.check_call(['rpm', '--import', key])
125
126
127def _run_yum_command(cmd, fatal=False):
128 """
129 Run an YUM command, checking output and retrying if the fatal flag is set
130 to True.
131
132 :param: cmd: str: The yum command to run.
133 :param: fatal: bool: Whether the command's output should be checked and
134 retried.
135 """
136 env = os.environ.copy()
137
138 if fatal:
139 retry_count = 0
140 result = None
141
142 # If the command is considered "fatal", we need to retry if the yum
143 # lock was not acquired.
144
145 while result is None or result == YUM_NO_LOCK:
146 try:
147 result = subprocess.check_call(cmd, env=env)
148 except subprocess.CalledProcessError as e:
149 retry_count = retry_count + 1
150 if retry_count > YUM_NO_LOCK_RETRY_COUNT:
151 raise
152 result = e.returncode
153 log("Couldn't acquire YUM lock. Will retry in {} seconds."
154 "".format(YUM_NO_LOCK_RETRY_DELAY))
155 time.sleep(YUM_NO_LOCK_RETRY_DELAY)
156
157 else:
158 subprocess.call(cmd, env=env)
0159
=== modified file 'hooks/charmhelpers/fetch/giturl.py' (properties changed: -x to +x)
--- hooks/charmhelpers/fetch/giturl.py 2015-03-23 09:45:10 +0000
+++ hooks/charmhelpers/fetch/giturl.py 2016-04-29 11:54:41 +0000
@@ -15,24 +15,18 @@
15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.15# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1616
17import os17import os
18from subprocess import check_call, CalledProcessError
18from charmhelpers.fetch import (19from charmhelpers.fetch import (
19 BaseFetchHandler,20 BaseFetchHandler,
20 UnhandledSource21 UnhandledSource,
22 filter_installed_packages,
23 install,
21)24)
22from charmhelpers.core.host import mkdir25
2326if filter_installed_packages(['git']) != []:
24import six27 install(['git'])
25if six.PY3:28 if filter_installed_packages(['git']) != []:
26 raise ImportError('GitPython does not support Python 3')29 raise NotImplementedError('Unable to install git')
27
28try:
29 from git import Repo
30except ImportError:
31 from charmhelpers.fetch import apt_install
32 apt_install("python-git")
33 from git import Repo
34
35from git.exc import GitCommandError # noqa E402
3630
3731
38class GitUrlFetchHandler(BaseFetchHandler):32class GitUrlFetchHandler(BaseFetchHandler):
@@ -40,19 +34,26 @@
40 def can_handle(self, source):34 def can_handle(self, source):
41 url_parts = self.parse_url(source)35 url_parts = self.parse_url(source)
42 # TODO (mattyw) no support for ssh git@ yet36 # TODO (mattyw) no support for ssh git@ yet
43 if url_parts.scheme not in ('http', 'https', 'git'):37 if url_parts.scheme not in ('http', 'https', 'git', ''):
44 return False38 return False
39 elif not url_parts.scheme:
40 return os.path.exists(os.path.join(source, '.git'))
45 else:41 else:
46 return True42 return True
4743
48 def clone(self, source, dest, branch):44 def clone(self, source, dest, branch="master", depth=None):
49 if not self.can_handle(source):45 if not self.can_handle(source):
50 raise UnhandledSource("Cannot handle {}".format(source))46 raise UnhandledSource("Cannot handle {}".format(source))
5147
52 repo = Repo.clone_from(source, dest)48 if os.path.exists(dest):
53 repo.git.checkout(branch)49 cmd = ['git', '-C', dest, 'pull', source, branch]
50 else:
51 cmd = ['git', 'clone', source, dest, '--branch', branch]
52 if depth:
53 cmd.extend(['--depth', depth])
54 check_call(cmd)
5455
55 def install(self, source, branch="master", dest=None):56 def install(self, source, branch="master", dest=None, depth=None):
56 url_parts = self.parse_url(source)57 url_parts = self.parse_url(source)
57 branch_name = url_parts.path.strip("/").split("/")[-1]58 branch_name = url_parts.path.strip("/").split("/")[-1]
58 if dest:59 if dest:
@@ -60,12 +61,10 @@
60 else:61 else:
61 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",62 dest_dir = os.path.join(os.environ.get('CHARM_DIR'), "fetched",
62 branch_name)63 branch_name)
63 if not os.path.exists(dest_dir):
64 mkdir(dest_dir, perms=0o755)
65 try:64 try:
66 self.clone(source, dest_dir, branch)65 self.clone(source, dest_dir, branch, depth)
67 except GitCommandError as e:66 except CalledProcessError as e:
68 raise UnhandledSource(e.message)67 raise UnhandledSource(e)
69 except OSError as e:68 except OSError as e:
70 raise UnhandledSource(e.strerror)69 raise UnhandledSource(e.strerror)
71 return dest_dir70 return dest_dir
7271
=== added directory 'hooks/charmhelpers/fetch/ubuntu'
=== added file 'hooks/charmhelpers/fetch/ubuntu/__init__.py'
--- hooks/charmhelpers/fetch/ubuntu/__init__.py 1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/fetch/ubuntu/__init__.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,296 @@
1import os
2import six
3import time
4import subprocess
5
6from tempfile import NamedTemporaryFile
7from charmhelpers.core.host import (
8 lsb_release
9)
10from charmhelpers.core.hookenv import (
11 log,
12)
13from charmhelpers.fetch import (
14 SourceConfigError,
15)
16
17CLOUD_ARCHIVE = """# Ubuntu Cloud Archive
18deb http://ubuntu-cloud.archive.canonical.com/ubuntu {} main
19"""
20PROPOSED_POCKET = """# Proposed
21deb http://archive.ubuntu.com/ubuntu {}-proposed main universe multiverse restricted
22"""
23CLOUD_ARCHIVE_POCKETS = {
24 # Folsom
25 'folsom': 'precise-updates/folsom',
26 'precise-folsom': 'precise-updates/folsom',
27 'precise-folsom/updates': 'precise-updates/folsom',
28 'precise-updates/folsom': 'precise-updates/folsom',
29 'folsom/proposed': 'precise-proposed/folsom',
30 'precise-folsom/proposed': 'precise-proposed/folsom',
31 'precise-proposed/folsom': 'precise-proposed/folsom',
32 # Grizzly
33 'grizzly': 'precise-updates/grizzly',
34 'precise-grizzly': 'precise-updates/grizzly',
35 'precise-grizzly/updates': 'precise-updates/grizzly',
36 'precise-updates/grizzly': 'precise-updates/grizzly',
37 'grizzly/proposed': 'precise-proposed/grizzly',
38 'precise-grizzly/proposed': 'precise-proposed/grizzly',
39 'precise-proposed/grizzly': 'precise-proposed/grizzly',
40 # Havana
41 'havana': 'precise-updates/havana',
42 'precise-havana': 'precise-updates/havana',
43 'precise-havana/updates': 'precise-updates/havana',
44 'precise-updates/havana': 'precise-updates/havana',
45 'havana/proposed': 'precise-proposed/havana',
46 'precise-havana/proposed': 'precise-proposed/havana',
47 'precise-proposed/havana': 'precise-proposed/havana',
48 # Icehouse
49 'icehouse': 'precise-updates/icehouse',
50 'precise-icehouse': 'precise-updates/icehouse',
51 'precise-icehouse/updates': 'precise-updates/icehouse',
52 'precise-updates/icehouse': 'precise-updates/icehouse',
53 'icehouse/proposed': 'precise-proposed/icehouse',
54 'precise-icehouse/proposed': 'precise-proposed/icehouse',
55 'precise-proposed/icehouse': 'precise-proposed/icehouse',
56 # Juno
57 'juno': 'trusty-updates/juno',
58 'trusty-juno': 'trusty-updates/juno',
59 'trusty-juno/updates': 'trusty-updates/juno',
60 'trusty-updates/juno': 'trusty-updates/juno',
61 'juno/proposed': 'trusty-proposed/juno',
62 'trusty-juno/proposed': 'trusty-proposed/juno',
63 'trusty-proposed/juno': 'trusty-proposed/juno',
64 # Kilo
65 'kilo': 'trusty-updates/kilo',
66 'trusty-kilo': 'trusty-updates/kilo',
67 'trusty-kilo/updates': 'trusty-updates/kilo',
68 'trusty-updates/kilo': 'trusty-updates/kilo',
69 'kilo/proposed': 'trusty-proposed/kilo',
70 'trusty-kilo/proposed': 'trusty-proposed/kilo',
71 'trusty-proposed/kilo': 'trusty-proposed/kilo',
72 # Liberty
73 'liberty': 'trusty-updates/liberty',
74 'trusty-liberty': 'trusty-updates/liberty',
75 'trusty-liberty/updates': 'trusty-updates/liberty',
76 'trusty-updates/liberty': 'trusty-updates/liberty',
77 'liberty/proposed': 'trusty-proposed/liberty',
78 'trusty-liberty/proposed': 'trusty-proposed/liberty',
79 'trusty-proposed/liberty': 'trusty-proposed/liberty',
80 # Mitaka
81 'mitaka': 'trusty-updates/mitaka',
82 'trusty-mitaka': 'trusty-updates/mitaka',
83 'trusty-mitaka/updates': 'trusty-updates/mitaka',
84 'trusty-updates/mitaka': 'trusty-updates/mitaka',
85 'mitaka/proposed': 'trusty-proposed/mitaka',
86 'trusty-mitaka/proposed': 'trusty-proposed/mitaka',
87 'trusty-proposed/mitaka': 'trusty-proposed/mitaka',
88}
89
90
91APT_NO_LOCK = 100 # The return code for "couldn't acquire lock" in APT.
92APT_NO_LOCK_RETRY_DELAY = 10 # Wait 10 seconds between apt lock checks.
93APT_NO_LOCK_RETRY_COUNT = 30 # Retry to acquire the lock X times.
94
95
96def filter_installed_packages(packages):
97 """Returns a list of packages that require installation"""
98 temp_cache = apt_cache()
99 _pkgs = []
100 for package in packages:
101 try:
102 p = temp_cache[package]
103 p.current_ver or _pkgs.append(package)
104 except KeyError:
105 log('Package {} has no installation candidate.'.format(package),
106 level='WARNING')
107 _pkgs.append(package)
108 return _pkgs
109
110
111def apt_cache(in_memory=True):
112 """Build and return an apt cache"""
113 from apt import apt_pkg
114 apt_pkg.init()
115 if in_memory:
116 apt_pkg.config.set("Dir::Cache::pkgcache", "")
117 apt_pkg.config.set("Dir::Cache::srcpkgcache", "")
118 return apt_pkg.Cache()
119
120
121def install(packages, options=None, fatal=False):
122 """Install one or more packages"""
123 if options is None:
124 options = ['--option=Dpkg::Options::=--force-confold']
125
126 cmd = ['apt-get', '--assume-yes']
127 cmd.extend(options)
128 cmd.append('install')
129 if isinstance(packages, six.string_types):
130 cmd.append(packages)
131 else:
132 cmd.extend(packages)
133 log("Installing {} with options: {}".format(packages,
134 options))
135 _run_apt_command(cmd, fatal)
136
137
138def upgrade(options=None, fatal=False, dist=False):
139 """Upgrade all packages"""
140 if options is None:
141 options = ['--option=Dpkg::Options::=--force-confold']
142
143 cmd = ['apt-get', '--assume-yes']
144 cmd.extend(options)
145 if dist:
146 cmd.append('dist-upgrade')
147 else:
148 cmd.append('upgrade')
149 log("Upgrading with options: {}".format(options))
150 _run_apt_command(cmd, fatal)
151
152
153def update(fatal=False):
154 """Update local apt cache"""
155 cmd = ['apt-get', 'update']
156 _run_apt_command(cmd, fatal)
157
158
159def purge(packages, fatal=False):
160 """Purge one or more packages"""
161 cmd = ['apt-get', '--assume-yes', 'purge']
162 if isinstance(packages, six.string_types):
163 cmd.append(packages)
164 else:
165 cmd.extend(packages)
166 log("Purging {}".format(packages))
167 _run_apt_command(cmd, fatal)
168
169
170def apt_mark(packages, mark, fatal=False):
171 """Flag one or more packages using apt-mark"""
172 log("Marking {} as {}".format(packages, mark))
173 cmd = ['apt-mark', mark]
174 if isinstance(packages, six.string_types):
175 cmd.append(packages)
176 else:
177 cmd.extend(packages)
178
179 if fatal:
180 subprocess.check_call(cmd, universal_newlines=True)
181 else:
182 subprocess.call(cmd, universal_newlines=True)
183
184
185def apt_hold(packages, fatal=False):
186 return apt_mark(packages, 'hold', fatal=fatal)
187
188
189def apt_unhold(packages, fatal=False):
190 return apt_mark(packages, 'unhold', fatal=fatal)
191
192
193def add_source(source, key=None):
194 """Add a package source to this system.
195
196 @param source: a URL or sources.list entry, as supported by
197 add-apt-repository(1). Examples::
198
199 ppa:charmers/example
200 deb https://stub:key@private.example.com/ubuntu trusty main
201
202 In addition:
203 'proposed:' may be used to enable the standard 'proposed'
204 pocket for the release.
205 'cloud:' may be used to activate official cloud archive pockets,
206 such as 'cloud:icehouse'
207 'distro' may be used as a noop
208
209 @param key: A key to be added to the system's APT keyring and used
210 to verify the signatures on packages. Ideally, this should be an
211 ASCII format GPG public key including the block headers. A GPG key
212 id may also be used, but be aware that only insecure protocols are
213 available to retrieve the actual public key from a public keyserver
214 placing your Juju environment at risk. ppa and cloud archive keys
215 are securely added automtically, so sould not be provided.
216 """
217 if source is None:
218 log('Source is not present. Skipping')
219 return
220
221 if (source.startswith('ppa:') or
222 source.startswith('http') or
223 source.startswith('deb ') or
224 source.startswith('cloud-archive:')):
225 subprocess.check_call(['add-apt-repository', '--yes', source])
226 elif source.startswith('cloud:'):
227 install(filter_installed_packages(['ubuntu-cloud-keyring']),
228 fatal=True)
229 pocket = source.split(':')[-1]
230 if pocket not in CLOUD_ARCHIVE_POCKETS:
231 raise SourceConfigError(
232 'Unsupported cloud: source option %s' %
233 pocket)
234 actual_pocket = CLOUD_ARCHIVE_POCKETS[pocket]
235 with open('/etc/apt/sources.list.d/cloud-archive.list', 'w') as apt:
236 apt.write(CLOUD_ARCHIVE.format(actual_pocket))
237 elif source == 'proposed':
238 release = lsb_release()['DISTRIB_CODENAME']
239 with open('/etc/apt/sources.list.d/proposed.list', 'w') as apt:
240 apt.write(PROPOSED_POCKET.format(release))
241 elif source == 'distro':
242 pass
243 else:
244 log("Unknown source: {!r}".format(source))
245
246 if key:
247 if '-----BEGIN PGP PUBLIC KEY BLOCK-----' in key:
248 with NamedTemporaryFile('w+') as key_file:
249 key_file.write(key)
250 key_file.flush()
251 key_file.seek(0)
252 subprocess.check_call(['apt-key', 'add', '-'], stdin=key_file)
253 else:
254 # Note that hkp: is in no way a secure protocol. Using a
255 # GPG key id is pointless from a security POV unless you
256 # absolutely trust your network and DNS.
257 subprocess.check_call(['apt-key', 'adv', '--keyserver',
258 'hkp://keyserver.ubuntu.com:80', '--recv',
259 key])
260
261
262def _run_apt_command(cmd, fatal=False):
263 """
264 Run an APT command, checking output and retrying if the fatal flag is set
265 to True.
266
267 :param: cmd: str: The apt command to run.
268 :param: fatal: bool: Whether the command's output should be checked and
269 retried.
270 """
271 env = os.environ.copy()
272
273 if 'DEBIAN_FRONTEND' not in env:
274 env['DEBIAN_FRONTEND'] = 'noninteractive'
275
276 if fatal:
277 retry_count = 0
278 result = None
279
280 # If the command is considered "fatal", we need to retry if the apt
281 # lock was not acquired.
282
283 while result is None or result == APT_NO_LOCK:
284 try:
285 result = subprocess.check_call(cmd, env=env)
286 except subprocess.CalledProcessError as e:
287 retry_count = retry_count + 1
288 if retry_count > APT_NO_LOCK_RETRY_COUNT:
289 raise
290 result = e.returncode
291 log("Couldn't acquire DPKG lock. Will retry in {} seconds."
292 "".format(APT_NO_LOCK_RETRY_DELAY))
293 time.sleep(APT_NO_LOCK_RETRY_DELAY)
294
295 else:
296 subprocess.call(cmd, env=env)
0297
=== modified file 'hooks/nrpe_helpers.py' (properties changed: -x to +x)
=== modified file 'hooks/nrpe_utils.py' (properties changed: -x to +x)
--- hooks/nrpe_utils.py 2016-01-18 22:54:35 +0000
+++ hooks/nrpe_utils.py 2016-04-29 11:54:41 +0000
@@ -1,14 +1,18 @@
1import os1import os
2import shutil2import shutil
3import glob3import glob
4import importlib
45
5from charmhelpers import fetch6from charmhelpers import fetch
6from charmhelpers.core import host7from charmhelpers.core import host
7from charmhelpers.core.templating import render8from charmhelpers.core.templating import render
8from charmhelpers.core import hookenv9from charmhelpers.core import hookenv
10from charmhelpers import get_platform
911
10import nrpe_helpers12import nrpe_helpers
1113
14platform = importlib.import_module(get_platform())
15
1216
13def restart_rsync(service_name):17def restart_rsync(service_name):
14 """ Restart rsync """18 """ Restart rsync """
@@ -17,25 +21,18 @@
1721
18def restart_nrpe(service_name):22def restart_nrpe(service_name):
19 """ Restart nrpe """23 """ Restart nrpe """
20 host.service_restart('nagios-nrpe-server')24 platform.restart_nrpe(service_name)
2125
2226
23def determine_packages():27def determine_packages():
24 """ List of packages this charm needs installed """28 """ List of packages this charm needs installed """
25 pkgs = [29 return platform.determine_packages()
26 'nagios-nrpe-server',
27 'nagios-plugins-basic',
28 'nagios-plugins-standard'
29 ]
30 if hookenv.config('export_nagios_definitions'):
31 pkgs.append('rsync')
32 return pkgs
3330
3431
35def install_packages(service_name):32def install_packages(service_name):
36 """ Install packages """33 """ Install packages """
37 fetch.apt_update()34 fetch.update()
38 fetch.apt_install(determine_packages(), fatal=True)35 fetch.install(determine_packages(), fatal=True)
3936
4037
41def remove_host_export_fragments(service_name):38def remove_host_export_fragments(service_name):
@@ -139,5 +136,6 @@
139 for rid in hookenv.relation_ids('monitors'):136 for rid in hookenv.relation_ids('monitors'):
140 hookenv.relation_set(137 hookenv.relation_set(
141 relation_id=rid,138 relation_id=rid,
142 relation_settings=monitor_relation.provide_data()139 relation_settings=monitor_relation.provide_data(),
140 charm_platform=get_platform()
143 )141 )
144142
=== modified file 'hooks/services.py' (properties changed: -x to +x)
--- hooks/services.py 2016-01-19 19:29:15 +0000
+++ hooks/services.py 2016-04-29 11:54:41 +0000
@@ -1,9 +1,13 @@
1import nrpe_utils
2import nrpe_helpers
3import importlib
4
1from charmhelpers.core import hookenv5from charmhelpers.core import hookenv
2from charmhelpers.core.services.base import ServiceManager6from charmhelpers.core.services.base import ServiceManager
3from charmhelpers.core.services import helpers7from charmhelpers.core.services import helpers
8from charmhelpers import get_platform
49
5import nrpe_utils10platform = importlib.import_module(get_platform())
6import nrpe_helpers
711
812
9def manage():13def manage():
@@ -29,10 +33,7 @@
29 nrpe_utils.update_nrpe_external_master_relation,33 nrpe_utils.update_nrpe_external_master_relation,
30 nrpe_utils.update_monitor_relation,34 nrpe_utils.update_monitor_relation,
31 nrpe_utils.render_nrped_files,35 nrpe_utils.render_nrped_files,
32 helpers.render_template(36 platform.render_nrpe_template(),
33 source='nrpe.tmpl',
34 target='/etc/nagios/nrpe.cfg'
35 ),
36 ],37 ],
37 'provided_data': [nrpe_helpers.PrincipleRelation()],38 'provided_data': [nrpe_helpers.PrincipleRelation()],
38 'start': [nrpe_utils.restart_nrpe],39 'start': [nrpe_utils.restart_nrpe],
3940
=== added file 'hooks/ubuntu.py'
--- hooks/ubuntu.py 1970-01-01 00:00:00 +0000
+++ hooks/ubuntu.py 2016-04-29 11:54:41 +0000
@@ -0,0 +1,27 @@
1from charmhelpers.core import host
2from charmhelpers.core import hookenv
3from charmhelpers.core.services import helpers
4
5
6def determine_packages():
7 """ List of packages this charm needs installed """
8 pkgs = [
9 'nagios-nrpe-server',
10 'nagios-plugins-basic',
11 'nagios-plugins-standard'
12 ]
13 if hookenv.config('export_nagios_definitions'):
14 pkgs.append('rsync')
15 return pkgs
16
17
18def restart_nrpe(service_name):
19 """ Restart nrpe """
20 host.service_restart('nagios-nrpe-server')
21
22
23def render_nrpe_template():
24 return helpers.render_template(
25 source='nrpe.tmpl',
26 target='/etc/nagios/nrpe.cfg'
27 )
028
=== added file 'templates/nrpe-centos.tmpl'
--- templates/nrpe-centos.tmpl 1970-01-01 00:00:00 +0000
+++ templates/nrpe-centos.tmpl 2016-04-29 11:54:41 +0000
@@ -0,0 +1,16 @@
1#--------------------------------------------------------
2# This file is managed by Juju
3#--------------------------------------------------------
4
5server_port={{ server_port }}
6allowed_hosts={{ external_nagios_master }},{{ monitor_allowed_hosts }}
7nrpe_user=nrpe
8nrpe_group=nrpe
9dont_blame_nrpe=0cat
10debug=0
11command_timeout=60
12pid_file=/var/run/nrpe/nrpe.pid
13
14# All configuration snippets go into nrpe.d/
15include_dir=/etc/nagios/nrpe.d/
16
017
=== modified file 'tests/11-monitors-configurations'
--- tests/11-monitors-configurations 2016-02-03 22:17:17 +0000
+++ tests/11-monitors-configurations 2016-04-29 11:54:41 +0000
@@ -37,11 +37,11 @@
37 # look for procrunning in nrpe config37 # look for procrunning in nrpe config
38 try:38 try:
39 mysql_unit.file_contents('/etc/nagios/nrpe.d/'39 mysql_unit.file_contents('/etc/nagios/nrpe.d/'
40 'check_proc_mysqld_principle.cfg')40 'check_total_procs_testing.cfg')
41 except IOError as e:41 except IOError as e:
42 amulet.raise_status(amulet.ERROR,42 amulet.raise_status(amulet.FAIL,
43 msg="procrunning config not found. Error:" +43 msg="procrunning config not found. Error: {0}".format(e)
44 e.args[1])44 )
4545
4646
47def test_nagios_monitors_response():47def test_nagios_monitors_response():
@@ -52,7 +52,7 @@
52 r = requests.get(host_url % nagios_unit.info['public-address'],52 r = requests.get(host_url % nagios_unit.info['public-address'],
53 auth=('nagiosadmin', nagpwd))53 auth=('nagiosadmin', nagpwd))
54 if not r.text.find('mysql-0-basic'):54 if not r.text.find('mysql-0-basic'):
55 amulet.raise_status(amulet.ERROR,55 amulet.raise_status(amulet.FAIL,
56 msg='Nagios is not monitoring the' +56 msg='Nagios is not monitoring the' +
57 ' hosts it supposed to.')57 ' hosts it supposed to.')
5858
5959
=== modified file 'tests/13-monitors-config'
--- tests/13-monitors-config 2016-02-03 22:17:17 +0000
+++ tests/13-monitors-config 2016-04-29 11:54:41 +0000
@@ -49,11 +49,11 @@
49 # look for procrunning in nrpe config49 # look for procrunning in nrpe config
50 try:50 try:
51 mysql_unit.file_contents('/etc/nagios/nrpe.d/'51 mysql_unit.file_contents('/etc/nagios/nrpe.d/'
52 'check_proc_mysqld_principle.cfg')52 'check_total_procs_testing.cfg')
53 except IOError as e:53 except IOError as e:
54 amulet.raise_status(amulet.ERROR,54 amulet.raise_status(amulet.FAIL,
55 msg="procrunning config not found. Error:" +55 msg="procrunning config not found. Error: {0}".format(e)
56 e.args[1])56 )
5757
5858
59def test_nagios_monitors_response():59def test_nagios_monitors_response():
@@ -64,7 +64,7 @@
64 r = requests.get(host_url % nagios_unit.info['public-address'],64 r = requests.get(host_url % nagios_unit.info['public-address'],
65 auth=('nagiosadmin', nagpwd))65 auth=('nagiosadmin', nagpwd))
66 if not r.text.find('processcount'):66 if not r.text.find('processcount'):
67 amulet.raise_status(amulet.ERROR,67 amulet.raise_status(amulet.FAIL,
68 msg='Nagios is not monitoring the' +68 msg='Nagios is not monitoring the' +
69 ' hosts it supposed to.')69 ' hosts it supposed to.')
7070

Subscribers

People subscribed via source and target branches

to all changes: