Merge lp:~gnuoy/charms/trusty/odl-controller/new-tests into lp:~sdn-charmers/charms/trusty/odl-controller/trunk

Proposed by Liam Young on 2015-11-11
Status: Merged
Merged at revision: 11
Proposed branch: lp:~gnuoy/charms/trusty/odl-controller/new-tests
Merge into: lp:~sdn-charmers/charms/trusty/odl-controller/trunk
Diff against target: 7239 lines (+5974/-195)
48 files modified
Makefile (+17/-0)
charm-helpers-sync.yaml (+2/-0)
charm-helpers-tests.yaml (+5/-0)
config.yaml (+4/-2)
hooks/charmhelpers/contrib/network/__init__.py (+15/-0)
hooks/charmhelpers/contrib/network/ip.py (+456/-0)
hooks/charmhelpers/contrib/openstack/amulet/deployment.py (+124/-11)
hooks/charmhelpers/contrib/openstack/amulet/utils.py (+381/-0)
hooks/charmhelpers/contrib/openstack/context.py (+169/-55)
hooks/charmhelpers/contrib/openstack/neutron.py (+57/-16)
hooks/charmhelpers/contrib/openstack/templates/ceph.conf (+6/-0)
hooks/charmhelpers/contrib/openstack/templating.py (+32/-4)
hooks/charmhelpers/contrib/openstack/utils.py (+313/-21)
hooks/charmhelpers/contrib/python/__init__.py (+15/-0)
hooks/charmhelpers/contrib/python/packages.py (+121/-0)
hooks/charmhelpers/contrib/storage/linux/ceph.py (+226/-13)
hooks/charmhelpers/contrib/storage/linux/utils.py (+4/-3)
hooks/charmhelpers/core/files.py (+45/-0)
hooks/charmhelpers/core/hookenv.py (+157/-14)
hooks/charmhelpers/core/host.py (+147/-28)
hooks/charmhelpers/core/hugepage.py (+71/-0)
hooks/charmhelpers/core/kernel.py (+68/-0)
hooks/charmhelpers/core/services/helpers.py (+22/-3)
hooks/charmhelpers/core/strutils.py (+30/-0)
hooks/charmhelpers/core/templating.py (+13/-6)
hooks/charmhelpers/core/unitdata.py (+61/-17)
hooks/charmhelpers/fetch/__init__.py (+9/-1)
metadata.yaml (+1/-1)
tests/015-basic-trusty-icehouse (+9/-0)
tests/016-basic-trusty-juno (+11/-0)
tests/017-basic-trusty-kilo (+11/-0)
tests/018-basic-trusty-liberty (+11/-0)
tests/basic_deployment.py (+465/-0)
tests/charmhelpers/__init__.py (+38/-0)
tests/charmhelpers/contrib/__init__.py (+15/-0)
tests/charmhelpers/contrib/amulet/__init__.py (+15/-0)
tests/charmhelpers/contrib/amulet/deployment.py (+95/-0)
tests/charmhelpers/contrib/amulet/utils.py (+818/-0)
tests/charmhelpers/contrib/openstack/__init__.py (+15/-0)
tests/charmhelpers/contrib/openstack/amulet/__init__.py (+15/-0)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+297/-0)
tests/charmhelpers/contrib/openstack/amulet/utils.py (+985/-0)
tests/setup/00-setup (+17/-0)
unit_tests/__init__.py (+3/-0)
unit_tests/odl_outputs.py (+271/-0)
unit_tests/test_odl_controller_hooks.py (+95/-0)
unit_tests/test_odl_controller_utils.py (+93/-0)
unit_tests/test_utils.py (+124/-0)
To merge this branch: bzr merge lp:~gnuoy/charms/trusty/odl-controller/new-tests
Reviewer Review Type Date Requested Status
James Page 2015-11-11 Needs Fixing on 2015-11-11
Review via email: mp+277258@code.launchpad.net
To post a comment you must log in.
James Page (james-page) wrote :

Hi Liam

Branch generally looks OK but a few niggles

1) The Makefile does not pass through the AMULET env variable for the karaf URL - I hacked this in for testing but please do update.

2) trusty-icehouse works OK; however juno and kilo both failed with:

2015-11-11 17:17:33 Starting deployment of devel3
2015-11-11 17:18:04 Invalid config charm openvswitch-odl openstack-origin=cloud:trusty-juno
2015-11-11 17:18:04 Invalid config charm neutron-api-odl openstack-origin=cloud:trusty-juno
2015-11-11 17:18:04 Invalid config charm /home/ubuntu/charms/trusty/odl-controller openstack-origin=cloud:trusty-juno
2015-11-11 17:18:04 Deployment stopped. run time: 31.35

I think those two charms need adding to the 'no_origin' list in charm-helpers - maybe we need a way to extend that list based on the test being executed so we don't have to update charm-helpers all of the time.

I'm assuming the juno was pre-decomposition of the odl mechanism driver; can we add liberty as well please?

review: Needs Fixing
12. By Liam Young on 2015-11-11

Fixes

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile'
2--- Makefile 2015-02-19 22:08:13 +0000
3+++ Makefile 2015-11-11 19:55:10 +0000
4@@ -1,6 +1,22 @@
5 #!/usr/bin/make
6 PYTHON := /usr/bin/env python
7
8+lint:
9+ @flake8 --exclude hooks/charmhelpers,tests/charmhelpers \
10+ hooks unit_tests tests
11+ @charm proof
12+
13+test:
14+ @# Bundletester expects unit tests here.
15+ @echo Starting unit tests...
16+ @$(PYTHON) /usr/bin/nosetests -v --nologcapture --with-coverage unit_tests
17+
18+functional_test:
19+ @echo Starting amulet tests...
20+ @tests/setup/00-setup
21+ @juju test -v -p AMULET_ODL_LOCATION,AMULET_HTTP_PROXY,AMULET_OS_VIP \
22+ --timeout 2700
23+
24 bin/charm_helpers_sync.py:
25 @mkdir -p bin
26 @bzr cat lp:charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py \
27@@ -8,3 +24,4 @@
28
29 sync: bin/charm_helpers_sync.py
30 @$(PYTHON) bin/charm_helpers_sync.py -c charm-helpers-sync.yaml
31+ @$(PYTHON) bin/charm_helpers_sync.py -c charm-helpers-tests.yaml
32
33=== modified file 'charm-helpers-sync.yaml'
34--- charm-helpers-sync.yaml 2015-02-19 22:08:13 +0000
35+++ charm-helpers-sync.yaml 2015-11-11 19:55:10 +0000
36@@ -6,3 +6,5 @@
37 - payload
38 - contrib.openstack|inc=*
39 - contrib.storage
40+ - contrib.network.ip
41+ - contrib.python.packages
42
43=== added file 'charm-helpers-tests.yaml'
44--- charm-helpers-tests.yaml 1970-01-01 00:00:00 +0000
45+++ charm-helpers-tests.yaml 2015-11-11 19:55:10 +0000
46@@ -0,0 +1,5 @@
47+branch: lp:charm-helpers
48+destination: tests/charmhelpers
49+include:
50+ - contrib.amulet
51+ - contrib.openstack.amulet
52
53=== modified file 'config.yaml'
54--- config.yaml 2015-11-04 19:14:40 +0000
55+++ config.yaml 2015-11-11 19:55:10 +0000
56@@ -19,17 +19,19 @@
57 package.
58 install-sources:
59 type: string
60+ default: ''
61 description: |
62 Package sources to install. Can be used to specify where to install the
63 opendaylight-karaf package from.
64 install-keys:
65 type: string
66+ default: ''
67 description: Apt keys for package install sources
68 http-proxy:
69 type: string
70- default:
71+ default: ''
72 description: Proxy to use for http connections for OpenDayLight
73 https-proxy:
74 type: string
75- default:
76+ default: ''
77 description: Proxy to use for https connections for OpenDayLight
78
79=== added directory 'hooks/charmhelpers/contrib/network'
80=== added file 'hooks/charmhelpers/contrib/network/__init__.py'
81--- hooks/charmhelpers/contrib/network/__init__.py 1970-01-01 00:00:00 +0000
82+++ hooks/charmhelpers/contrib/network/__init__.py 2015-11-11 19:55:10 +0000
83@@ -0,0 +1,15 @@
84+# Copyright 2014-2015 Canonical Limited.
85+#
86+# This file is part of charm-helpers.
87+#
88+# charm-helpers is free software: you can redistribute it and/or modify
89+# it under the terms of the GNU Lesser General Public License version 3 as
90+# published by the Free Software Foundation.
91+#
92+# charm-helpers is distributed in the hope that it will be useful,
93+# but WITHOUT ANY WARRANTY; without even the implied warranty of
94+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
95+# GNU Lesser General Public License for more details.
96+#
97+# You should have received a copy of the GNU Lesser General Public License
98+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
99
100=== added file 'hooks/charmhelpers/contrib/network/ip.py'
101--- hooks/charmhelpers/contrib/network/ip.py 1970-01-01 00:00:00 +0000
102+++ hooks/charmhelpers/contrib/network/ip.py 2015-11-11 19:55:10 +0000
103@@ -0,0 +1,456 @@
104+# Copyright 2014-2015 Canonical Limited.
105+#
106+# This file is part of charm-helpers.
107+#
108+# charm-helpers is free software: you can redistribute it and/or modify
109+# it under the terms of the GNU Lesser General Public License version 3 as
110+# published by the Free Software Foundation.
111+#
112+# charm-helpers is distributed in the hope that it will be useful,
113+# but WITHOUT ANY WARRANTY; without even the implied warranty of
114+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
115+# GNU Lesser General Public License for more details.
116+#
117+# You should have received a copy of the GNU Lesser General Public License
118+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
119+
120+import glob
121+import re
122+import subprocess
123+import six
124+import socket
125+
126+from functools import partial
127+
128+from charmhelpers.core.hookenv import unit_get
129+from charmhelpers.fetch import apt_install, apt_update
130+from charmhelpers.core.hookenv import (
131+ log,
132+ WARNING,
133+)
134+
135+try:
136+ import netifaces
137+except ImportError:
138+ apt_update(fatal=True)
139+ apt_install('python-netifaces', fatal=True)
140+ import netifaces
141+
142+try:
143+ import netaddr
144+except ImportError:
145+ apt_update(fatal=True)
146+ apt_install('python-netaddr', fatal=True)
147+ import netaddr
148+
149+
150+def _validate_cidr(network):
151+ try:
152+ netaddr.IPNetwork(network)
153+ except (netaddr.core.AddrFormatError, ValueError):
154+ raise ValueError("Network (%s) is not in CIDR presentation format" %
155+ network)
156+
157+
158+def no_ip_found_error_out(network):
159+ errmsg = ("No IP address found in network: %s" % network)
160+ raise ValueError(errmsg)
161+
162+
163+def get_address_in_network(network, fallback=None, fatal=False):
164+ """Get an IPv4 or IPv6 address within the network from the host.
165+
166+ :param network (str): CIDR presentation format. For example,
167+ '192.168.1.0/24'.
168+ :param fallback (str): If no address is found, return fallback.
169+ :param fatal (boolean): If no address is found, fallback is not
170+ set and fatal is True then exit(1).
171+ """
172+ if network is None:
173+ if fallback is not None:
174+ return fallback
175+
176+ if fatal:
177+ no_ip_found_error_out(network)
178+ else:
179+ return None
180+
181+ _validate_cidr(network)
182+ network = netaddr.IPNetwork(network)
183+ for iface in netifaces.interfaces():
184+ addresses = netifaces.ifaddresses(iface)
185+ if network.version == 4 and netifaces.AF_INET in addresses:
186+ addr = addresses[netifaces.AF_INET][0]['addr']
187+ netmask = addresses[netifaces.AF_INET][0]['netmask']
188+ cidr = netaddr.IPNetwork("%s/%s" % (addr, netmask))
189+ if cidr in network:
190+ return str(cidr.ip)
191+
192+ if network.version == 6 and netifaces.AF_INET6 in addresses:
193+ for addr in addresses[netifaces.AF_INET6]:
194+ if not addr['addr'].startswith('fe80'):
195+ cidr = netaddr.IPNetwork("%s/%s" % (addr['addr'],
196+ addr['netmask']))
197+ if cidr in network:
198+ return str(cidr.ip)
199+
200+ if fallback is not None:
201+ return fallback
202+
203+ if fatal:
204+ no_ip_found_error_out(network)
205+
206+ return None
207+
208+
209+def is_ipv6(address):
210+ """Determine whether provided address is IPv6 or not."""
211+ try:
212+ address = netaddr.IPAddress(address)
213+ except netaddr.AddrFormatError:
214+ # probably a hostname - so not an address at all!
215+ return False
216+
217+ return address.version == 6
218+
219+
220+def is_address_in_network(network, address):
221+ """
222+ Determine whether the provided address is within a network range.
223+
224+ :param network (str): CIDR presentation format. For example,
225+ '192.168.1.0/24'.
226+ :param address: An individual IPv4 or IPv6 address without a net
227+ mask or subnet prefix. For example, '192.168.1.1'.
228+ :returns boolean: Flag indicating whether address is in network.
229+ """
230+ try:
231+ network = netaddr.IPNetwork(network)
232+ except (netaddr.core.AddrFormatError, ValueError):
233+ raise ValueError("Network (%s) is not in CIDR presentation format" %
234+ network)
235+
236+ try:
237+ address = netaddr.IPAddress(address)
238+ except (netaddr.core.AddrFormatError, ValueError):
239+ raise ValueError("Address (%s) is not in correct presentation format" %
240+ address)
241+
242+ if address in network:
243+ return True
244+ else:
245+ return False
246+
247+
248+def _get_for_address(address, key):
249+ """Retrieve an attribute of or the physical interface that
250+ the IP address provided could be bound to.
251+
252+ :param address (str): An individual IPv4 or IPv6 address without a net
253+ mask or subnet prefix. For example, '192.168.1.1'.
254+ :param key: 'iface' for the physical interface name or an attribute
255+ of the configured interface, for example 'netmask'.
256+ :returns str: Requested attribute or None if address is not bindable.
257+ """
258+ address = netaddr.IPAddress(address)
259+ for iface in netifaces.interfaces():
260+ addresses = netifaces.ifaddresses(iface)
261+ if address.version == 4 and netifaces.AF_INET in addresses:
262+ addr = addresses[netifaces.AF_INET][0]['addr']
263+ netmask = addresses[netifaces.AF_INET][0]['netmask']
264+ network = netaddr.IPNetwork("%s/%s" % (addr, netmask))
265+ cidr = network.cidr
266+ if address in cidr:
267+ if key == 'iface':
268+ return iface
269+ else:
270+ return addresses[netifaces.AF_INET][0][key]
271+
272+ if address.version == 6 and netifaces.AF_INET6 in addresses:
273+ for addr in addresses[netifaces.AF_INET6]:
274+ if not addr['addr'].startswith('fe80'):
275+ network = netaddr.IPNetwork("%s/%s" % (addr['addr'],
276+ addr['netmask']))
277+ cidr = network.cidr
278+ if address in cidr:
279+ if key == 'iface':
280+ return iface
281+ elif key == 'netmask' and cidr:
282+ return str(cidr).split('/')[1]
283+ else:
284+ return addr[key]
285+
286+ return None
287+
288+
289+get_iface_for_address = partial(_get_for_address, key='iface')
290+
291+
292+get_netmask_for_address = partial(_get_for_address, key='netmask')
293+
294+
295+def format_ipv6_addr(address):
296+ """If address is IPv6, wrap it in '[]' otherwise return None.
297+
298+ This is required by most configuration files when specifying IPv6
299+ addresses.
300+ """
301+ if is_ipv6(address):
302+ return "[%s]" % address
303+
304+ return None
305+
306+
307+def get_iface_addr(iface='eth0', inet_type='AF_INET', inc_aliases=False,
308+ fatal=True, exc_list=None):
309+ """Return the assigned IP address for a given interface, if any."""
310+ # Extract nic if passed /dev/ethX
311+ if '/' in iface:
312+ iface = iface.split('/')[-1]
313+
314+ if not exc_list:
315+ exc_list = []
316+
317+ try:
318+ inet_num = getattr(netifaces, inet_type)
319+ except AttributeError:
320+ raise Exception("Unknown inet type '%s'" % str(inet_type))
321+
322+ interfaces = netifaces.interfaces()
323+ if inc_aliases:
324+ ifaces = []
325+ for _iface in interfaces:
326+ if iface == _iface or _iface.split(':')[0] == iface:
327+ ifaces.append(_iface)
328+
329+ if fatal and not ifaces:
330+ raise Exception("Invalid interface '%s'" % iface)
331+
332+ ifaces.sort()
333+ else:
334+ if iface not in interfaces:
335+ if fatal:
336+ raise Exception("Interface '%s' not found " % (iface))
337+ else:
338+ return []
339+
340+ else:
341+ ifaces = [iface]
342+
343+ addresses = []
344+ for netiface in ifaces:
345+ net_info = netifaces.ifaddresses(netiface)
346+ if inet_num in net_info:
347+ for entry in net_info[inet_num]:
348+ if 'addr' in entry and entry['addr'] not in exc_list:
349+ addresses.append(entry['addr'])
350+
351+ if fatal and not addresses:
352+ raise Exception("Interface '%s' doesn't have any %s addresses." %
353+ (iface, inet_type))
354+
355+ return sorted(addresses)
356+
357+
358+get_ipv4_addr = partial(get_iface_addr, inet_type='AF_INET')
359+
360+
361+def get_iface_from_addr(addr):
362+ """Work out on which interface the provided address is configured."""
363+ for iface in netifaces.interfaces():
364+ addresses = netifaces.ifaddresses(iface)
365+ for inet_type in addresses:
366+ for _addr in addresses[inet_type]:
367+ _addr = _addr['addr']
368+ # link local
369+ ll_key = re.compile("(.+)%.*")
370+ raw = re.match(ll_key, _addr)
371+ if raw:
372+ _addr = raw.group(1)
373+
374+ if _addr == addr:
375+ log("Address '%s' is configured on iface '%s'" %
376+ (addr, iface))
377+ return iface
378+
379+ msg = "Unable to infer net iface on which '%s' is configured" % (addr)
380+ raise Exception(msg)
381+
382+
383+def sniff_iface(f):
384+ """Ensure decorated function is called with a value for iface.
385+
386+ If no iface provided, inject net iface inferred from unit private address.
387+ """
388+ def iface_sniffer(*args, **kwargs):
389+ if not kwargs.get('iface', None):
390+ kwargs['iface'] = get_iface_from_addr(unit_get('private-address'))
391+
392+ return f(*args, **kwargs)
393+
394+ return iface_sniffer
395+
396+
397+@sniff_iface
398+def get_ipv6_addr(iface=None, inc_aliases=False, fatal=True, exc_list=None,
399+ dynamic_only=True):
400+ """Get assigned IPv6 address for a given interface.
401+
402+ Returns list of addresses found. If no address found, returns empty list.
403+
404+ If iface is None, we infer the current primary interface by doing a reverse
405+ lookup on the unit private-address.
406+
407+ We currently only support scope global IPv6 addresses i.e. non-temporary
408+ addresses. If no global IPv6 address is found, return the first one found
409+ in the ipv6 address list.
410+ """
411+ addresses = get_iface_addr(iface=iface, inet_type='AF_INET6',
412+ inc_aliases=inc_aliases, fatal=fatal,
413+ exc_list=exc_list)
414+
415+ if addresses:
416+ global_addrs = []
417+ for addr in addresses:
418+ key_scope_link_local = re.compile("^fe80::..(.+)%(.+)")
419+ m = re.match(key_scope_link_local, addr)
420+ if m:
421+ eui_64_mac = m.group(1)
422+ iface = m.group(2)
423+ else:
424+ global_addrs.append(addr)
425+
426+ if global_addrs:
427+ # Make sure any found global addresses are not temporary
428+ cmd = ['ip', 'addr', 'show', iface]
429+ out = subprocess.check_output(cmd).decode('UTF-8')
430+ if dynamic_only:
431+ key = re.compile("inet6 (.+)/[0-9]+ scope global dynamic.*")
432+ else:
433+ key = re.compile("inet6 (.+)/[0-9]+ scope global.*")
434+
435+ addrs = []
436+ for line in out.split('\n'):
437+ line = line.strip()
438+ m = re.match(key, line)
439+ if m and 'temporary' not in line:
440+ # Return the first valid address we find
441+ for addr in global_addrs:
442+ if m.group(1) == addr:
443+ if not dynamic_only or \
444+ m.group(1).endswith(eui_64_mac):
445+ addrs.append(addr)
446+
447+ if addrs:
448+ return addrs
449+
450+ if fatal:
451+ raise Exception("Interface '%s' does not have a scope global "
452+ "non-temporary ipv6 address." % iface)
453+
454+ return []
455+
456+
457+def get_bridges(vnic_dir='/sys/devices/virtual/net'):
458+ """Return a list of bridges on the system."""
459+ b_regex = "%s/*/bridge" % vnic_dir
460+ return [x.replace(vnic_dir, '').split('/')[1] for x in glob.glob(b_regex)]
461+
462+
463+def get_bridge_nics(bridge, vnic_dir='/sys/devices/virtual/net'):
464+ """Return a list of nics comprising a given bridge on the system."""
465+ brif_regex = "%s/%s/brif/*" % (vnic_dir, bridge)
466+ return [x.split('/')[-1] for x in glob.glob(brif_regex)]
467+
468+
469+def is_bridge_member(nic):
470+ """Check if a given nic is a member of a bridge."""
471+ for bridge in get_bridges():
472+ if nic in get_bridge_nics(bridge):
473+ return True
474+
475+ return False
476+
477+
478+def is_ip(address):
479+ """
480+ Returns True if address is a valid IP address.
481+ """
482+ try:
483+ # Test to see if already an IPv4 address
484+ socket.inet_aton(address)
485+ return True
486+ except socket.error:
487+ return False
488+
489+
490+def ns_query(address):
491+ try:
492+ import dns.resolver
493+ except ImportError:
494+ apt_install('python-dnspython')
495+ import dns.resolver
496+
497+ if isinstance(address, dns.name.Name):
498+ rtype = 'PTR'
499+ elif isinstance(address, six.string_types):
500+ rtype = 'A'
501+ else:
502+ return None
503+
504+ answers = dns.resolver.query(address, rtype)
505+ if answers:
506+ return str(answers[0])
507+ return None
508+
509+
510+def get_host_ip(hostname, fallback=None):
511+ """
512+ Resolves the IP for a given hostname, or returns
513+ the input if it is already an IP.
514+ """
515+ if is_ip(hostname):
516+ return hostname
517+
518+ ip_addr = ns_query(hostname)
519+ if not ip_addr:
520+ try:
521+ ip_addr = socket.gethostbyname(hostname)
522+ except:
523+ log("Failed to resolve hostname '%s'" % (hostname),
524+ level=WARNING)
525+ return fallback
526+ return ip_addr
527+
528+
529+def get_hostname(address, fqdn=True):
530+ """
531+ Resolves hostname for given IP, or returns the input
532+ if it is already a hostname.
533+ """
534+ if is_ip(address):
535+ try:
536+ import dns.reversename
537+ except ImportError:
538+ apt_install("python-dnspython")
539+ import dns.reversename
540+
541+ rev = dns.reversename.from_address(address)
542+ result = ns_query(rev)
543+
544+ if not result:
545+ try:
546+ result = socket.gethostbyaddr(address)[0]
547+ except:
548+ return None
549+ else:
550+ result = address
551+
552+ if fqdn:
553+ # strip trailing .
554+ if result.endswith('.'):
555+ return result[:-1]
556+ else:
557+ return result
558+ else:
559+ return result.split('.')[0]
560
561=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/deployment.py'
562--- hooks/charmhelpers/contrib/openstack/amulet/deployment.py 2015-07-22 12:10:31 +0000
563+++ hooks/charmhelpers/contrib/openstack/amulet/deployment.py 2015-11-11 19:55:10 +0000
564@@ -14,12 +14,18 @@
565 # You should have received a copy of the GNU Lesser General Public License
566 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
567
568+import logging
569+import re
570+import sys
571 import six
572 from collections import OrderedDict
573 from charmhelpers.contrib.amulet.deployment import (
574 AmuletDeployment
575 )
576
577+DEBUG = logging.DEBUG
578+ERROR = logging.ERROR
579+
580
581 class OpenStackAmuletDeployment(AmuletDeployment):
582 """OpenStack amulet deployment.
583@@ -28,9 +34,12 @@
584 that is specifically for use by OpenStack charms.
585 """
586
587- def __init__(self, series=None, openstack=None, source=None, stable=True):
588+ def __init__(self, series=None, openstack=None, source=None,
589+ stable=True, log_level=DEBUG):
590 """Initialize the deployment environment."""
591 super(OpenStackAmuletDeployment, self).__init__(series)
592+ self.log = self.get_logger(level=log_level)
593+ self.log.info('OpenStackAmuletDeployment: init')
594 self.openstack = openstack
595 self.source = source
596 self.stable = stable
597@@ -38,26 +47,55 @@
598 # out.
599 self.current_next = "trusty"
600
601+ def get_logger(self, name="deployment-logger", level=logging.DEBUG):
602+ """Get a logger object that will log to stdout."""
603+ log = logging
604+ logger = log.getLogger(name)
605+ fmt = log.Formatter("%(asctime)s %(funcName)s "
606+ "%(levelname)s: %(message)s")
607+
608+ handler = log.StreamHandler(stream=sys.stdout)
609+ handler.setLevel(level)
610+ handler.setFormatter(fmt)
611+
612+ logger.addHandler(handler)
613+ logger.setLevel(level)
614+
615+ return logger
616+
617 def _determine_branch_locations(self, other_services):
618 """Determine the branch locations for the other services.
619
620 Determine if the local branch being tested is derived from its
621 stable or next (dev) branch, and based on this, use the corresonding
622 stable or next branches for the other_services."""
623- base_charms = ['mysql', 'mongodb']
624+
625+ self.log.info('OpenStackAmuletDeployment: determine branch locations')
626+
627+ # Charms outside the lp:~openstack-charmers namespace
628+ base_charms = ['mysql', 'mongodb', 'nrpe']
629+
630+ # Force these charms to current series even when using an older series.
631+ # ie. Use trusty/nrpe even when series is precise, as the P charm
632+ # does not possess the necessary external master config and hooks.
633+ force_series_current = ['nrpe']
634
635 if self.series in ['precise', 'trusty']:
636 base_series = self.series
637 else:
638 base_series = self.current_next
639
640- if self.stable:
641- for svc in other_services:
642+ for svc in other_services:
643+ if svc['name'] in force_series_current:
644+ base_series = self.current_next
645+ # If a location has been explicitly set, use it
646+ if svc.get('location'):
647+ continue
648+ if self.stable:
649 temp = 'lp:charms/{}/{}'
650 svc['location'] = temp.format(base_series,
651 svc['name'])
652- else:
653- for svc in other_services:
654+ else:
655 if svc['name'] in base_charms:
656 temp = 'lp:charms/{}/{}'
657 svc['location'] = temp.format(base_series,
658@@ -66,10 +104,13 @@
659 temp = 'lp:~openstack-charmers/charms/{}/{}/next'
660 svc['location'] = temp.format(self.current_next,
661 svc['name'])
662+
663 return other_services
664
665 def _add_services(self, this_service, other_services):
666 """Add services to the deployment and set openstack-origin/source."""
667+ self.log.info('OpenStackAmuletDeployment: adding services')
668+
669 other_services = self._determine_branch_locations(other_services)
670
671 super(OpenStackAmuletDeployment, self)._add_services(this_service,
672@@ -77,29 +118,101 @@
673
674 services = other_services
675 services.append(this_service)
676+
677+ # Charms which should use the source config option
678 use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph',
679 'ceph-osd', 'ceph-radosgw']
680- # Most OpenStack subordinate charms do not expose an origin option
681- # as that is controlled by the principle.
682- ignore = ['cinder-ceph', 'hacluster', 'neutron-openvswitch']
683+
684+ # Charms which can not use openstack-origin, ie. many subordinates
685+ no_origin = ['cinder-ceph', 'hacluster', 'neutron-openvswitch', 'nrpe']
686
687 if self.openstack:
688 for svc in services:
689- if svc['name'] not in use_source + ignore:
690+ if svc['name'] not in use_source + no_origin:
691 config = {'openstack-origin': self.openstack}
692 self.d.configure(svc['name'], config)
693
694 if self.source:
695 for svc in services:
696- if svc['name'] in use_source and svc['name'] not in ignore:
697+ if svc['name'] in use_source and svc['name'] not in no_origin:
698 config = {'source': self.source}
699 self.d.configure(svc['name'], config)
700
701 def _configure_services(self, configs):
702 """Configure all of the services."""
703+ self.log.info('OpenStackAmuletDeployment: configure services')
704 for service, config in six.iteritems(configs):
705 self.d.configure(service, config)
706
707+ def _auto_wait_for_status(self, message=None, exclude_services=None,
708+ include_only=None, timeout=1800):
709+ """Wait for all units to have a specific extended status, except
710+ for any defined as excluded. Unless specified via message, any
711+ status containing any case of 'ready' will be considered a match.
712+
713+ Examples of message usage:
714+
715+ Wait for all unit status to CONTAIN any case of 'ready' or 'ok':
716+ message = re.compile('.*ready.*|.*ok.*', re.IGNORECASE)
717+
718+ Wait for all units to reach this status (exact match):
719+ message = re.compile('^Unit is ready and clustered$')
720+
721+ Wait for all units to reach any one of these (exact match):
722+ message = re.compile('Unit is ready|OK|Ready')
723+
724+ Wait for at least one unit to reach this status (exact match):
725+ message = {'ready'}
726+
727+ See Amulet's sentry.wait_for_messages() for message usage detail.
728+ https://github.com/juju/amulet/blob/master/amulet/sentry.py
729+
730+ :param message: Expected status match
731+ :param exclude_services: List of juju service names to ignore,
732+ not to be used in conjuction with include_only.
733+ :param include_only: List of juju service names to exclusively check,
734+ not to be used in conjuction with exclude_services.
735+ :param timeout: Maximum time in seconds to wait for status match
736+ :returns: None. Raises if timeout is hit.
737+ """
738+ self.log.info('Waiting for extended status on units...')
739+
740+ all_services = self.d.services.keys()
741+
742+ if exclude_services and include_only:
743+ raise ValueError('exclude_services can not be used '
744+ 'with include_only')
745+
746+ if message:
747+ if isinstance(message, re._pattern_type):
748+ match = message.pattern
749+ else:
750+ match = message
751+
752+ self.log.debug('Custom extended status wait match: '
753+ '{}'.format(match))
754+ else:
755+ self.log.debug('Default extended status wait match: contains '
756+ 'READY (case-insensitive)')
757+ message = re.compile('.*ready.*', re.IGNORECASE)
758+
759+ if exclude_services:
760+ self.log.debug('Excluding services from extended status match: '
761+ '{}'.format(exclude_services))
762+ else:
763+ exclude_services = []
764+
765+ if include_only:
766+ services = include_only
767+ else:
768+ services = list(set(all_services) - set(exclude_services))
769+
770+ self.log.debug('Waiting up to {}s for extended status on services: '
771+ '{}'.format(timeout, services))
772+ service_messages = {service: message for service in services}
773+ self.d.sentry.wait_for_messages(service_messages, timeout=timeout)
774+ self.log.info('OK')
775+
776 def _get_openstack_release(self):
777 """Get openstack release.
778
779
780=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/utils.py'
781--- hooks/charmhelpers/contrib/openstack/amulet/utils.py 2015-07-22 12:10:31 +0000
782+++ hooks/charmhelpers/contrib/openstack/amulet/utils.py 2015-11-11 19:55:10 +0000
783@@ -18,6 +18,7 @@
784 import json
785 import logging
786 import os
787+import re
788 import six
789 import time
790 import urllib
791@@ -27,6 +28,7 @@
792 import heatclient.v1.client as heat_client
793 import keystoneclient.v2_0 as keystone_client
794 import novaclient.v1_1.client as nova_client
795+import pika
796 import swiftclient
797
798 from charmhelpers.contrib.amulet.utils import (
799@@ -602,3 +604,382 @@
800 self.log.debug('Ceph {} samples (OK): '
801 '{}'.format(sample_type, samples))
802 return None
803+
804+ # rabbitmq/amqp specific helpers:
805+
806+ def rmq_wait_for_cluster(self, deployment, init_sleep=15, timeout=1200):
807+ """Wait for rmq units extended status to show cluster readiness,
808+ after an optional initial sleep period. Initial sleep is likely
809+ necessary to be effective following a config change, as status
810+ message may not instantly update to non-ready."""
811+
812+ if init_sleep:
813+ time.sleep(init_sleep)
814+
815+ message = re.compile('^Unit is ready and clustered$')
816+ deployment._auto_wait_for_status(message=message,
817+ timeout=timeout,
818+ include_only=['rabbitmq-server'])
819+
820+ def add_rmq_test_user(self, sentry_units,
821+ username="testuser1", password="changeme"):
822+ """Add a test user via the first rmq juju unit, check connection as
823+ the new user against all sentry units.
824+
825+ :param sentry_units: list of sentry unit pointers
826+ :param username: amqp user name, default to testuser1
827+ :param password: amqp user password
828+ :returns: None if successful. Raise on error.
829+ """
830+ self.log.debug('Adding rmq user ({})...'.format(username))
831+
832+ # Check that user does not already exist
833+ cmd_user_list = 'rabbitmqctl list_users'
834+ output, _ = self.run_cmd_unit(sentry_units[0], cmd_user_list)
835+ if username in output:
836+ self.log.warning('User ({}) already exists, returning '
837+ 'gracefully.'.format(username))
838+ return
839+
840+ perms = '".*" ".*" ".*"'
841+ cmds = ['rabbitmqctl add_user {} {}'.format(username, password),
842+ 'rabbitmqctl set_permissions {} {}'.format(username, perms)]
843+
844+ # Add user via first unit
845+ for cmd in cmds:
846+ output, _ = self.run_cmd_unit(sentry_units[0], cmd)
847+
848+ # Check connection against the other sentry_units
849+ self.log.debug('Checking user connect against units...')
850+ for sentry_unit in sentry_units:
851+ connection = self.connect_amqp_by_unit(sentry_unit, ssl=False,
852+ username=username,
853+ password=password)
854+ connection.close()
855+
856+ def delete_rmq_test_user(self, sentry_units, username="testuser1"):
857+ """Delete a rabbitmq user via the first rmq juju unit.
858+
859+ :param sentry_units: list of sentry unit pointers
860+ :param username: amqp user name, default to testuser1
861+ :param password: amqp user password
862+ :returns: None if successful or no such user.
863+ """
864+ self.log.debug('Deleting rmq user ({})...'.format(username))
865+
866+ # Check that the user exists
867+ cmd_user_list = 'rabbitmqctl list_users'
868+ output, _ = self.run_cmd_unit(sentry_units[0], cmd_user_list)
869+
870+ if username not in output:
871+ self.log.warning('User ({}) does not exist, returning '
872+ 'gracefully.'.format(username))
873+ return
874+
875+ # Delete the user
876+ cmd_user_del = 'rabbitmqctl delete_user {}'.format(username)
877+ output, _ = self.run_cmd_unit(sentry_units[0], cmd_user_del)
878+
879+ def get_rmq_cluster_status(self, sentry_unit):
880+ """Execute rabbitmq cluster status command on a unit and return
881+ the full output.
882+
883+ :param unit: sentry unit
884+ :returns: String containing console output of cluster status command
885+ """
886+ cmd = 'rabbitmqctl cluster_status'
887+ output, _ = self.run_cmd_unit(sentry_unit, cmd)
888+ self.log.debug('{} cluster_status:\n{}'.format(
889+ sentry_unit.info['unit_name'], output))
890+ return str(output)
891+
892+ def get_rmq_cluster_running_nodes(self, sentry_unit):
893+ """Parse rabbitmqctl cluster_status output string, return list of
894+ running rabbitmq cluster nodes.
895+
896+ :param unit: sentry unit
897+ :returns: List containing node names of running nodes
898+ """
899+ # NOTE(beisner): rabbitmqctl cluster_status output is not
900+ # json-parsable, do string chop foo, then json.loads that.
901+ str_stat = self.get_rmq_cluster_status(sentry_unit)
902+ if 'running_nodes' in str_stat:
903+ pos_start = str_stat.find("{running_nodes,") + 15
904+ pos_end = str_stat.find("]},", pos_start) + 1
905+ str_run_nodes = str_stat[pos_start:pos_end].replace("'", '"')
906+ run_nodes = json.loads(str_run_nodes)
907+ return run_nodes
908+ else:
909+ return []
910+
911+ def validate_rmq_cluster_running_nodes(self, sentry_units):
912+ """Check that all rmq unit hostnames are represented in the
913+ cluster_status output of all units.
914+
915+ :param host_names: dict of juju unit names to host names
916+ :param units: list of sentry unit pointers (all rmq units)
917+ :returns: None if successful, otherwise return error message
918+ """
919+ host_names = self.get_unit_hostnames(sentry_units)
920+ errors = []
921+
922+ # Query every unit for cluster_status running nodes
923+ for query_unit in sentry_units:
924+ query_unit_name = query_unit.info['unit_name']
925+ running_nodes = self.get_rmq_cluster_running_nodes(query_unit)
926+
927+ # Confirm that every unit is represented in the queried unit's
928+ # cluster_status running nodes output.
929+ for validate_unit in sentry_units:
930+ val_host_name = host_names[validate_unit.info['unit_name']]
931+ val_node_name = 'rabbit@{}'.format(val_host_name)
932+
933+ if val_node_name not in running_nodes:
934+ errors.append('Cluster member check failed on {}: {} not '
935+ 'in {}\n'.format(query_unit_name,
936+ val_node_name,
937+ running_nodes))
938+ if errors:
939+ return ''.join(errors)
940+
941+ def rmq_ssl_is_enabled_on_unit(self, sentry_unit, port=None):
942+ """Check a single juju rmq unit for ssl and port in the config file."""
943+ host = sentry_unit.info['public-address']
944+ unit_name = sentry_unit.info['unit_name']
945+
946+ conf_file = '/etc/rabbitmq/rabbitmq.config'
947+ conf_contents = str(self.file_contents_safe(sentry_unit,
948+ conf_file, max_wait=16))
949+ # Checks
950+ conf_ssl = 'ssl' in conf_contents
951+ conf_port = str(port) in conf_contents
952+
953+ # Port explicitly checked in config
954+ if port and conf_port and conf_ssl:
955+ self.log.debug('SSL is enabled @{}:{} '
956+ '({})'.format(host, port, unit_name))
957+ return True
958+ elif port and not conf_port and conf_ssl:
959+ self.log.debug('SSL is enabled @{} but not on port {} '
960+ '({})'.format(host, port, unit_name))
961+ return False
962+ # Port not checked (useful when checking that ssl is disabled)
963+ elif not port and conf_ssl:
964+ self.log.debug('SSL is enabled @{}:{} '
965+ '({})'.format(host, port, unit_name))
966+ return True
967+ elif not conf_ssl:
968+ self.log.debug('SSL not enabled @{}:{} '
969+ '({})'.format(host, port, unit_name))
970+ return False
971+ else:
972+ msg = ('Unknown condition when checking SSL status @{}:{} '
973+ '({})'.format(host, port, unit_name))
974+ amulet.raise_status(amulet.FAIL, msg)
975+
976+ def validate_rmq_ssl_enabled_units(self, sentry_units, port=None):
977+ """Check that ssl is enabled on rmq juju sentry units.
978+
979+ :param sentry_units: list of all rmq sentry units
980+ :param port: optional ssl port override to validate
981+ :returns: None if successful, otherwise return error message
982+ """
983+ for sentry_unit in sentry_units:
984+ if not self.rmq_ssl_is_enabled_on_unit(sentry_unit, port=port):
985+ return ('Unexpected condition: ssl is disabled on unit '
986+ '({})'.format(sentry_unit.info['unit_name']))
987+ return None
988+
989+ def validate_rmq_ssl_disabled_units(self, sentry_units):
990+ """Check that ssl is enabled on listed rmq juju sentry units.
991+
992+ :param sentry_units: list of all rmq sentry units
993+ :returns: True if successful. Raise on error.
994+ """
995+ for sentry_unit in sentry_units:
996+ if self.rmq_ssl_is_enabled_on_unit(sentry_unit):
997+ return ('Unexpected condition: ssl is enabled on unit '
998+ '({})'.format(sentry_unit.info['unit_name']))
999+ return None
1000+
1001+ def configure_rmq_ssl_on(self, sentry_units, deployment,
1002+ port=None, max_wait=60):
1003+ """Turn ssl charm config option on, with optional non-default
1004+ ssl port specification. Confirm that it is enabled on every
1005+ unit.
1006+
1007+ :param sentry_units: list of sentry units
1008+ :param deployment: amulet deployment object pointer
1009+ :param port: amqp port, use defaults if None
1010+ :param max_wait: maximum time to wait in seconds to confirm
1011+ :returns: None if successful. Raise on error.
1012+ """
1013+ self.log.debug('Setting ssl charm config option: on')
1014+
1015+ # Enable RMQ SSL
1016+ config = {'ssl': 'on'}
1017+ if port:
1018+ config['ssl_port'] = port
1019+
1020+ deployment.d.configure('rabbitmq-server', config)
1021+
1022+ # Wait for unit status
1023+ self.rmq_wait_for_cluster(deployment)
1024+
1025+ # Confirm
1026+ tries = 0
1027+ ret = self.validate_rmq_ssl_enabled_units(sentry_units, port=port)
1028+ while ret and tries < (max_wait / 4):
1029+ time.sleep(4)
1030+ self.log.debug('Attempt {}: {}'.format(tries, ret))
1031+ ret = self.validate_rmq_ssl_enabled_units(sentry_units, port=port)
1032+ tries += 1
1033+
1034+ if ret:
1035+ amulet.raise_status(amulet.FAIL, ret)
1036+
1037+ def configure_rmq_ssl_off(self, sentry_units, deployment, max_wait=60):
1038+ """Turn ssl charm config option off, confirm that it is disabled
1039+ on every unit.
1040+
1041+ :param sentry_units: list of sentry units
1042+ :param deployment: amulet deployment object pointer
1043+ :param max_wait: maximum time to wait in seconds to confirm
1044+ :returns: None if successful. Raise on error.
1045+ """
1046+ self.log.debug('Setting ssl charm config option: off')
1047+
1048+ # Disable RMQ SSL
1049+ config = {'ssl': 'off'}
1050+ deployment.d.configure('rabbitmq-server', config)
1051+
1052+ # Wait for unit status
1053+ self.rmq_wait_for_cluster(deployment)
1054+
1055+ # Confirm
1056+ tries = 0
1057+ ret = self.validate_rmq_ssl_disabled_units(sentry_units)
1058+ while ret and tries < (max_wait / 4):
1059+ time.sleep(4)
1060+ self.log.debug('Attempt {}: {}'.format(tries, ret))
1061+ ret = self.validate_rmq_ssl_disabled_units(sentry_units)
1062+ tries += 1
1063+
1064+ if ret:
1065+ amulet.raise_status(amulet.FAIL, ret)
1066+
1067+ def connect_amqp_by_unit(self, sentry_unit, ssl=False,
1068+ port=None, fatal=True,
1069+ username="testuser1", password="changeme"):
1070+ """Establish and return a pika amqp connection to the rabbitmq service
1071+ running on a rmq juju unit.
1072+
1073+ :param sentry_unit: sentry unit pointer
1074+ :param ssl: boolean, default to False
1075+ :param port: amqp port, use defaults if None
1076+ :param fatal: boolean, default to True (raises on connect error)
1077+ :param username: amqp user name, default to testuser1
1078+ :param password: amqp user password
1079+ :returns: pika amqp connection pointer or None if failed and non-fatal
1080+ """
1081+ host = sentry_unit.info['public-address']
1082+ unit_name = sentry_unit.info['unit_name']
1083+
1084+ # Default port logic if port is not specified
1085+ if ssl and not port:
1086+ port = 5671
1087+ elif not ssl and not port:
1088+ port = 5672
1089+
1090+ self.log.debug('Connecting to amqp on {}:{} ({}) as '
1091+ '{}...'.format(host, port, unit_name, username))
1092+
1093+ try:
1094+ credentials = pika.PlainCredentials(username, password)
1095+ parameters = pika.ConnectionParameters(host=host, port=port,
1096+ credentials=credentials,
1097+ ssl=ssl,
1098+ connection_attempts=3,
1099+ retry_delay=5,
1100+ socket_timeout=1)
1101+ connection = pika.BlockingConnection(parameters)
1102+ assert connection.server_properties['product'] == 'RabbitMQ'
1103+ self.log.debug('Connect OK')
1104+ return connection
1105+ except Exception as e:
1106+ msg = ('amqp connection failed to {}:{} as '
1107+ '{} ({})'.format(host, port, username, str(e)))
1108+ if fatal:
1109+ amulet.raise_status(amulet.FAIL, msg)
1110+ else:
1111+ self.log.warn(msg)
1112+ return None
1113+
1114+ def publish_amqp_message_by_unit(self, sentry_unit, message,
1115+ queue="test", ssl=False,
1116+ username="testuser1",
1117+ password="changeme",
1118+ port=None):
1119+ """Publish an amqp message to a rmq juju unit.
1120+
1121+ :param sentry_unit: sentry unit pointer
1122+ :param message: amqp message string
1123+ :param queue: message queue, default to test
1124+ :param username: amqp user name, default to testuser1
1125+ :param password: amqp user password
1126+ :param ssl: boolean, default to False
1127+ :param port: amqp port, use defaults if None
1128+ :returns: None. Raises exception if publish failed.
1129+ """
1130+ self.log.debug('Publishing message to {} queue:\n{}'.format(queue,
1131+ message))
1132+ connection = self.connect_amqp_by_unit(sentry_unit, ssl=ssl,
1133+ port=port,
1134+ username=username,
1135+ password=password)
1136+
1137+ # NOTE(beisner): extra debug here re: pika hang potential:
1138+ # https://github.com/pika/pika/issues/297
1139+ # https://groups.google.com/forum/#!topic/rabbitmq-users/Ja0iyfF0Szw
1140+ self.log.debug('Defining channel...')
1141+ channel = connection.channel()
1142+ self.log.debug('Declaring queue...')
1143+ channel.queue_declare(queue=queue, auto_delete=False, durable=True)
1144+ self.log.debug('Publishing message...')
1145+ channel.basic_publish(exchange='', routing_key=queue, body=message)
1146+ self.log.debug('Closing channel...')
1147+ channel.close()
1148+ self.log.debug('Closing connection...')
1149+ connection.close()
1150+
1151+ def get_amqp_message_by_unit(self, sentry_unit, queue="test",
1152+ username="testuser1",
1153+ password="changeme",
1154+ ssl=False, port=None):
1155+ """Get an amqp message from a rmq juju unit.
1156+
1157+ :param sentry_unit: sentry unit pointer
1158+ :param queue: message queue, default to test
1159+ :param username: amqp user name, default to testuser1
1160+ :param password: amqp user password
1161+ :param ssl: boolean, default to False
1162+ :param port: amqp port, use defaults if None
1163+ :returns: amqp message body as string. Raise if get fails.
1164+ """
1165+ connection = self.connect_amqp_by_unit(sentry_unit, ssl=ssl,
1166+ port=port,
1167+ username=username,
1168+ password=password)
1169+ channel = connection.channel()
1170+ method_frame, _, body = channel.basic_get(queue)
1171+
1172+ if method_frame:
1173+ self.log.debug('Retreived message from {} queue:\n{}'.format(queue,
1174+ body))
1175+ channel.basic_ack(method_frame.delivery_tag)
1176+ channel.close()
1177+ connection.close()
1178+ return body
1179+ else:
1180+ msg = 'No message retrieved.'
1181+ amulet.raise_status(amulet.FAIL, msg)
1182
1183=== modified file 'hooks/charmhelpers/contrib/openstack/context.py'
1184--- hooks/charmhelpers/contrib/openstack/context.py 2015-07-22 12:10:31 +0000
1185+++ hooks/charmhelpers/contrib/openstack/context.py 2015-11-11 19:55:10 +0000
1186@@ -14,6 +14,7 @@
1187 # You should have received a copy of the GNU Lesser General Public License
1188 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1189
1190+import glob
1191 import json
1192 import os
1193 import re
1194@@ -50,6 +51,8 @@
1195 from charmhelpers.core.strutils import bool_from_string
1196
1197 from charmhelpers.core.host import (
1198+ get_bond_master,
1199+ is_phy_iface,
1200 list_nics,
1201 get_nic_hwaddr,
1202 mkdir,
1203@@ -192,10 +195,50 @@
1204 class OSContextGenerator(object):
1205 """Base class for all context generators."""
1206 interfaces = []
1207+ related = False
1208+ complete = False
1209+ missing_data = []
1210
1211 def __call__(self):
1212 raise NotImplementedError
1213
1214+ def context_complete(self, ctxt):
1215+ """Check for missing data for the required context data.
1216+ Set self.missing_data if it exists and return False.
1217+ Set self.complete if no missing data and return True.
1218+ """
1219+ # Fresh start
1220+ self.complete = False
1221+ self.missing_data = []
1222+ for k, v in six.iteritems(ctxt):
1223+ if v is None or v == '':
1224+ if k not in self.missing_data:
1225+ self.missing_data.append(k)
1226+
1227+ if self.missing_data:
1228+ self.complete = False
1229+ log('Missing required data: %s' % ' '.join(self.missing_data), level=INFO)
1230+ else:
1231+ self.complete = True
1232+ return self.complete
1233+
1234+ def get_related(self):
1235+ """Check if any of the context interfaces have relation ids.
1236+ Set self.related and return True if one of the interfaces
1237+ has relation ids.
1238+ """
1239+ # Fresh start
1240+ self.related = False
1241+ try:
1242+ for interface in self.interfaces:
1243+ if relation_ids(interface):
1244+ self.related = True
1245+ return self.related
1246+ except AttributeError as e:
1247+ log("{} {}"
1248+ "".format(self, e), 'INFO')
1249+ return self.related
1250+
1251
1252 class SharedDBContext(OSContextGenerator):
1253 interfaces = ['shared-db']
1254@@ -211,6 +254,7 @@
1255 self.database = database
1256 self.user = user
1257 self.ssl_dir = ssl_dir
1258+ self.rel_name = self.interfaces[0]
1259
1260 def __call__(self):
1261 self.database = self.database or config('database')
1262@@ -244,6 +288,7 @@
1263 password_setting = self.relation_prefix + '_password'
1264
1265 for rid in relation_ids(self.interfaces[0]):
1266+ self.related = True
1267 for unit in related_units(rid):
1268 rdata = relation_get(rid=rid, unit=unit)
1269 host = rdata.get('db_host')
1270@@ -255,7 +300,7 @@
1271 'database_password': rdata.get(password_setting),
1272 'database_type': 'mysql'
1273 }
1274- if context_complete(ctxt):
1275+ if self.context_complete(ctxt):
1276 db_ssl(rdata, ctxt, self.ssl_dir)
1277 return ctxt
1278 return {}
1279@@ -276,6 +321,7 @@
1280
1281 ctxt = {}
1282 for rid in relation_ids(self.interfaces[0]):
1283+ self.related = True
1284 for unit in related_units(rid):
1285 rel_host = relation_get('host', rid=rid, unit=unit)
1286 rel_user = relation_get('user', rid=rid, unit=unit)
1287@@ -285,7 +331,7 @@
1288 'database_user': rel_user,
1289 'database_password': rel_passwd,
1290 'database_type': 'postgresql'}
1291- if context_complete(ctxt):
1292+ if self.context_complete(ctxt):
1293 return ctxt
1294
1295 return {}
1296@@ -346,6 +392,7 @@
1297 ctxt['signing_dir'] = cachedir
1298
1299 for rid in relation_ids(self.rel_name):
1300+ self.related = True
1301 for unit in related_units(rid):
1302 rdata = relation_get(rid=rid, unit=unit)
1303 serv_host = rdata.get('service_host')
1304@@ -364,7 +411,7 @@
1305 'service_protocol': svc_protocol,
1306 'auth_protocol': auth_protocol})
1307
1308- if context_complete(ctxt):
1309+ if self.context_complete(ctxt):
1310 # NOTE(jamespage) this is required for >= icehouse
1311 # so a missing value just indicates keystone needs
1312 # upgrading
1313@@ -403,6 +450,7 @@
1314 ctxt = {}
1315 for rid in relation_ids(self.rel_name):
1316 ha_vip_only = False
1317+ self.related = True
1318 for unit in related_units(rid):
1319 if relation_get('clustered', rid=rid, unit=unit):
1320 ctxt['clustered'] = True
1321@@ -435,7 +483,7 @@
1322 ha_vip_only = relation_get('ha-vip-only',
1323 rid=rid, unit=unit) is not None
1324
1325- if context_complete(ctxt):
1326+ if self.context_complete(ctxt):
1327 if 'rabbit_ssl_ca' in ctxt:
1328 if not self.ssl_dir:
1329 log("Charm not setup for ssl support but ssl ca "
1330@@ -467,7 +515,7 @@
1331 ctxt['oslo_messaging_flags'] = config_flags_parser(
1332 oslo_messaging_flags)
1333
1334- if not context_complete(ctxt):
1335+ if not self.complete:
1336 return {}
1337
1338 return ctxt
1339@@ -483,13 +531,15 @@
1340
1341 log('Generating template context for ceph', level=DEBUG)
1342 mon_hosts = []
1343- auth = None
1344- key = None
1345- use_syslog = str(config('use-syslog')).lower()
1346+ ctxt = {
1347+ 'use_syslog': str(config('use-syslog')).lower()
1348+ }
1349 for rid in relation_ids('ceph'):
1350 for unit in related_units(rid):
1351- auth = relation_get('auth', rid=rid, unit=unit)
1352- key = relation_get('key', rid=rid, unit=unit)
1353+ if not ctxt.get('auth'):
1354+ ctxt['auth'] = relation_get('auth', rid=rid, unit=unit)
1355+ if not ctxt.get('key'):
1356+ ctxt['key'] = relation_get('key', rid=rid, unit=unit)
1357 ceph_pub_addr = relation_get('ceph-public-address', rid=rid,
1358 unit=unit)
1359 unit_priv_addr = relation_get('private-address', rid=rid,
1360@@ -498,15 +548,12 @@
1361 ceph_addr = format_ipv6_addr(ceph_addr) or ceph_addr
1362 mon_hosts.append(ceph_addr)
1363
1364- ctxt = {'mon_hosts': ' '.join(sorted(mon_hosts)),
1365- 'auth': auth,
1366- 'key': key,
1367- 'use_syslog': use_syslog}
1368+ ctxt['mon_hosts'] = ' '.join(sorted(mon_hosts))
1369
1370 if not os.path.isdir('/etc/ceph'):
1371 os.mkdir('/etc/ceph')
1372
1373- if not context_complete(ctxt):
1374+ if not self.context_complete(ctxt):
1375 return {}
1376
1377 ensure_packages(['ceph-common'])
1378@@ -893,6 +940,31 @@
1379 'neutron_url': '%s://%s:%s' % (proto, host, '9696')}
1380 return ctxt
1381
1382+ def pg_ctxt(self):
1383+ driver = neutron_plugin_attribute(self.plugin, 'driver',
1384+ self.network_manager)
1385+ config = neutron_plugin_attribute(self.plugin, 'config',
1386+ self.network_manager)
1387+ ovs_ctxt = {'core_plugin': driver,
1388+ 'neutron_plugin': 'plumgrid',
1389+ 'neutron_security_groups': self.neutron_security_groups,
1390+ 'local_ip': unit_private_ip(),
1391+ 'config': config}
1392+ return ovs_ctxt
1393+
1394+ def midonet_ctxt(self):
1395+ driver = neutron_plugin_attribute(self.plugin, 'driver',
1396+ self.network_manager)
1397+ midonet_config = neutron_plugin_attribute(self.plugin, 'config',
1398+ self.network_manager)
1399+ mido_ctxt = {'core_plugin': driver,
1400+ 'neutron_plugin': 'midonet',
1401+ 'neutron_security_groups': self.neutron_security_groups,
1402+ 'local_ip': unit_private_ip(),
1403+ 'config': midonet_config}
1404+
1405+ return mido_ctxt
1406+
1407 def __call__(self):
1408 if self.network_manager not in ['quantum', 'neutron']:
1409 return {}
1410@@ -912,6 +984,10 @@
1411 ctxt.update(self.calico_ctxt())
1412 elif self.plugin == 'vsp':
1413 ctxt.update(self.nuage_ctxt())
1414+ elif self.plugin == 'plumgrid':
1415+ ctxt.update(self.pg_ctxt())
1416+ elif self.plugin == 'midonet':
1417+ ctxt.update(self.midonet_ctxt())
1418
1419 alchemy_flags = config('neutron-alchemy-flags')
1420 if alchemy_flags:
1421@@ -923,7 +999,6 @@
1422
1423
1424 class NeutronPortContext(OSContextGenerator):
1425- NIC_PREFIXES = ['eth', 'bond']
1426
1427 def resolve_ports(self, ports):
1428 """Resolve NICs not yet bound to bridge(s)
1429@@ -935,7 +1010,18 @@
1430
1431 hwaddr_to_nic = {}
1432 hwaddr_to_ip = {}
1433- for nic in list_nics(self.NIC_PREFIXES):
1434+ for nic in list_nics():
1435+ # Ignore virtual interfaces (bond masters will be identified from
1436+ # their slaves)
1437+ if not is_phy_iface(nic):
1438+ continue
1439+
1440+ _nic = get_bond_master(nic)
1441+ if _nic:
1442+ log("Replacing iface '%s' with bond master '%s'" % (nic, _nic),
1443+ level=DEBUG)
1444+ nic = _nic
1445+
1446 hwaddr = get_nic_hwaddr(nic)
1447 hwaddr_to_nic[hwaddr] = nic
1448 addresses = get_ipv4_addr(nic, fatal=False)
1449@@ -961,7 +1047,8 @@
1450 # trust it to be the real external network).
1451 resolved.append(entry)
1452
1453- return resolved
1454+ # Ensure no duplicates
1455+ return list(set(resolved))
1456
1457
1458 class OSConfigFlagContext(OSContextGenerator):
1459@@ -1033,7 +1120,7 @@
1460
1461 ctxt = {
1462 ... other context ...
1463- 'subordinate_config': {
1464+ 'subordinate_configuration': {
1465 'DEFAULT': {
1466 'key1': 'value1',
1467 },
1468@@ -1051,13 +1138,22 @@
1469 :param config_file : Service's config file to query sections
1470 :param interface : Subordinate interface to inspect
1471 """
1472- self.service = service
1473 self.config_file = config_file
1474- self.interface = interface
1475+ if isinstance(service, list):
1476+ self.services = service
1477+ else:
1478+ self.services = [service]
1479+ if isinstance(interface, list):
1480+ self.interfaces = interface
1481+ else:
1482+ self.interfaces = [interface]
1483
1484 def __call__(self):
1485 ctxt = {'sections': {}}
1486- for rid in relation_ids(self.interface):
1487+ rids = []
1488+ for interface in self.interfaces:
1489+ rids.extend(relation_ids(interface))
1490+ for rid in rids:
1491 for unit in related_units(rid):
1492 sub_config = relation_get('subordinate_configuration',
1493 rid=rid, unit=unit)
1494@@ -1065,33 +1161,37 @@
1495 try:
1496 sub_config = json.loads(sub_config)
1497 except:
1498- log('Could not parse JSON from subordinate_config '
1499- 'setting from %s' % rid, level=ERROR)
1500- continue
1501-
1502- if self.service not in sub_config:
1503- log('Found subordinate_config on %s but it contained'
1504- 'nothing for %s service' % (rid, self.service),
1505- level=INFO)
1506- continue
1507-
1508- sub_config = sub_config[self.service]
1509- if self.config_file not in sub_config:
1510- log('Found subordinate_config on %s but it contained'
1511- 'nothing for %s' % (rid, self.config_file),
1512- level=INFO)
1513- continue
1514-
1515- sub_config = sub_config[self.config_file]
1516- for k, v in six.iteritems(sub_config):
1517- if k == 'sections':
1518- for section, config_dict in six.iteritems(v):
1519- log("adding section '%s'" % (section),
1520- level=DEBUG)
1521- ctxt[k][section] = config_dict
1522- else:
1523- ctxt[k] = v
1524-
1525+ log('Could not parse JSON from '
1526+ 'subordinate_configuration setting from %s'
1527+ % rid, level=ERROR)
1528+ continue
1529+
1530+ for service in self.services:
1531+ if service not in sub_config:
1532+ log('Found subordinate_configuration on %s but it '
1533+ 'contained nothing for %s service'
1534+ % (rid, service), level=INFO)
1535+ continue
1536+
1537+ sub_config = sub_config[service]
1538+ if self.config_file not in sub_config:
1539+ log('Found subordinate_configuration on %s but it '
1540+ 'contained nothing for %s'
1541+ % (rid, self.config_file), level=INFO)
1542+ continue
1543+
1544+ sub_config = sub_config[self.config_file]
1545+ for k, v in six.iteritems(sub_config):
1546+ if k == 'sections':
1547+ for section, config_list in six.iteritems(v):
1548+ log("adding section '%s'" % (section),
1549+ level=DEBUG)
1550+ if ctxt[k].get(section):
1551+ ctxt[k][section].extend(config_list)
1552+ else:
1553+ ctxt[k][section] = config_list
1554+ else:
1555+ ctxt[k] = v
1556 log("%d section(s) found" % (len(ctxt['sections'])), level=DEBUG)
1557 return ctxt
1558
1559@@ -1268,15 +1368,19 @@
1560 def __call__(self):
1561 ports = config('data-port')
1562 if ports:
1563+ # Map of {port/mac:bridge}
1564 portmap = parse_data_port_mappings(ports)
1565- ports = portmap.values()
1566+ ports = portmap.keys()
1567+ # Resolve provided ports or mac addresses and filter out those
1568+ # already attached to a bridge.
1569 resolved = self.resolve_ports(ports)
1570+ # FIXME: is this necessary?
1571 normalized = {get_nic_hwaddr(port): port for port in resolved
1572 if port not in ports}
1573 normalized.update({port: port for port in resolved
1574 if port in ports})
1575 if resolved:
1576- return {bridge: normalized[port] for bridge, port in
1577+ return {normalized[port]: bridge for port, bridge in
1578 six.iteritems(portmap) if port in normalized.keys()}
1579
1580 return None
1581@@ -1287,12 +1391,22 @@
1582 def __call__(self):
1583 ctxt = {}
1584 mappings = super(PhyNICMTUContext, self).__call__()
1585- if mappings and mappings.values():
1586- ports = mappings.values()
1587+ if mappings and mappings.keys():
1588+ ports = sorted(mappings.keys())
1589 napi_settings = NeutronAPIContext()()
1590 mtu = napi_settings.get('network_device_mtu')
1591+ all_ports = set()
1592+ # If any of ports is a vlan device, its underlying device must have
1593+ # mtu applied first.
1594+ for port in ports:
1595+ for lport in glob.glob("/sys/class/net/%s/lower_*" % port):
1596+ lport = os.path.basename(lport)
1597+ all_ports.add(lport.split('_')[1])
1598+
1599+ all_ports = list(all_ports)
1600+ all_ports.extend(ports)
1601 if mtu:
1602- ctxt["devs"] = '\\n'.join(ports)
1603+ ctxt["devs"] = '\\n'.join(all_ports)
1604 ctxt['mtu'] = mtu
1605
1606 return ctxt
1607@@ -1324,6 +1438,6 @@
1608 'auth_protocol':
1609 rdata.get('auth_protocol') or 'http',
1610 }
1611- if context_complete(ctxt):
1612+ if self.context_complete(ctxt):
1613 return ctxt
1614 return {}
1615
1616=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
1617--- hooks/charmhelpers/contrib/openstack/neutron.py 2015-07-22 12:10:31 +0000
1618+++ hooks/charmhelpers/contrib/openstack/neutron.py 2015-11-11 19:55:10 +0000
1619@@ -195,6 +195,34 @@
1620 'packages': [],
1621 'server_packages': ['neutron-server', 'neutron-plugin-nuage'],
1622 'server_services': ['neutron-server']
1623+ },
1624+ 'plumgrid': {
1625+ 'config': '/etc/neutron/plugins/plumgrid/plumgrid.ini',
1626+ 'driver': 'neutron.plugins.plumgrid.plumgrid_plugin.plumgrid_plugin.NeutronPluginPLUMgridV2',
1627+ 'contexts': [
1628+ context.SharedDBContext(user=config('database-user'),
1629+ database=config('database'),
1630+ ssl_dir=NEUTRON_CONF_DIR)],
1631+ 'services': [],
1632+ 'packages': [['plumgrid-lxc'],
1633+ ['iovisor-dkms']],
1634+ 'server_packages': ['neutron-server',
1635+ 'neutron-plugin-plumgrid'],
1636+ 'server_services': ['neutron-server']
1637+ },
1638+ 'midonet': {
1639+ 'config': '/etc/neutron/plugins/midonet/midonet.ini',
1640+ 'driver': 'midonet.neutron.plugin.MidonetPluginV2',
1641+ 'contexts': [
1642+ context.SharedDBContext(user=config('neutron-database-user'),
1643+ database=config('neutron-database'),
1644+ relation_prefix='neutron',
1645+ ssl_dir=NEUTRON_CONF_DIR)],
1646+ 'services': [],
1647+ 'packages': [[headers_package()] + determine_dkms_package()],
1648+ 'server_packages': ['neutron-server',
1649+ 'python-neutron-plugin-midonet'],
1650+ 'server_services': ['neutron-server']
1651 }
1652 }
1653 if release >= 'icehouse':
1654@@ -255,17 +283,30 @@
1655 return 'neutron'
1656
1657
1658-def parse_mappings(mappings):
1659+def parse_mappings(mappings, key_rvalue=False):
1660+ """By default mappings are lvalue keyed.
1661+
1662+ If key_rvalue is True, the mapping will be reversed to allow multiple
1663+ configs for the same lvalue.
1664+ """
1665 parsed = {}
1666 if mappings:
1667 mappings = mappings.split()
1668 for m in mappings:
1669 p = m.partition(':')
1670- key = p[0].strip()
1671- if p[1]:
1672- parsed[key] = p[2].strip()
1673+
1674+ if key_rvalue:
1675+ key_index = 2
1676+ val_index = 0
1677+ # if there is no rvalue skip to next
1678+ if not p[1]:
1679+ continue
1680 else:
1681- parsed[key] = ''
1682+ key_index = 0
1683+ val_index = 2
1684+
1685+ key = p[key_index].strip()
1686+ parsed[key] = p[val_index].strip()
1687
1688 return parsed
1689
1690@@ -283,25 +324,25 @@
1691 def parse_data_port_mappings(mappings, default_bridge='br-data'):
1692 """Parse data port mappings.
1693
1694- Mappings must be a space-delimited list of bridge:port mappings.
1695+ Mappings must be a space-delimited list of bridge:port.
1696
1697- Returns dict of the form {bridge:port}.
1698+ Returns dict of the form {port:bridge} where ports may be mac addresses or
1699+ interface names.
1700 """
1701- _mappings = parse_mappings(mappings)
1702+
1703+ # NOTE(dosaboy): we use rvalue for key to allow multiple values to be
1704+ # proposed for <port> since it may be a mac address which will differ
1705+ # across units this allowing first-known-good to be chosen.
1706+ _mappings = parse_mappings(mappings, key_rvalue=True)
1707 if not _mappings or list(_mappings.values()) == ['']:
1708 if not mappings:
1709 return {}
1710
1711 # For backwards-compatibility we need to support port-only provided in
1712 # config.
1713- _mappings = {default_bridge: mappings.split()[0]}
1714-
1715- bridges = _mappings.keys()
1716- ports = _mappings.values()
1717- if len(set(bridges)) != len(bridges):
1718- raise Exception("It is not allowed to have more than one port "
1719- "configured on the same bridge")
1720-
1721+ _mappings = {mappings.split()[0]: default_bridge}
1722+
1723+ ports = _mappings.keys()
1724 if len(set(ports)) != len(ports):
1725 raise Exception("It is not allowed to have the same port configured "
1726 "on more than one bridge")
1727
1728=== modified file 'hooks/charmhelpers/contrib/openstack/templates/ceph.conf'
1729--- hooks/charmhelpers/contrib/openstack/templates/ceph.conf 2015-07-22 12:10:31 +0000
1730+++ hooks/charmhelpers/contrib/openstack/templates/ceph.conf 2015-11-11 19:55:10 +0000
1731@@ -13,3 +13,9 @@
1732 err to syslog = {{ use_syslog }}
1733 clog to syslog = {{ use_syslog }}
1734
1735+[client]
1736+{% if rbd_client_cache_settings -%}
1737+{% for key, value in rbd_client_cache_settings.iteritems() -%}
1738+{{ key }} = {{ value }}
1739+{% endfor -%}
1740+{%- endif %}
1741\ No newline at end of file
1742
1743=== modified file 'hooks/charmhelpers/contrib/openstack/templating.py'
1744--- hooks/charmhelpers/contrib/openstack/templating.py 2015-02-19 22:08:13 +0000
1745+++ hooks/charmhelpers/contrib/openstack/templating.py 2015-11-11 19:55:10 +0000
1746@@ -18,7 +18,7 @@
1747
1748 import six
1749
1750-from charmhelpers.fetch import apt_install
1751+from charmhelpers.fetch import apt_install, apt_update
1752 from charmhelpers.core.hookenv import (
1753 log,
1754 ERROR,
1755@@ -29,8 +29,9 @@
1756 try:
1757 from jinja2 import FileSystemLoader, ChoiceLoader, Environment, exceptions
1758 except ImportError:
1759- # python-jinja2 may not be installed yet, or we're running unittests.
1760- FileSystemLoader = ChoiceLoader = Environment = exceptions = None
1761+ apt_update(fatal=True)
1762+ apt_install('python-jinja2', fatal=True)
1763+ from jinja2 import FileSystemLoader, ChoiceLoader, Environment, exceptions
1764
1765
1766 class OSConfigException(Exception):
1767@@ -112,7 +113,7 @@
1768
1769 def complete_contexts(self):
1770 '''
1771- Return a list of interfaces that have atisfied contexts.
1772+ Return a list of interfaces that have satisfied contexts.
1773 '''
1774 if self._complete_contexts:
1775 return self._complete_contexts
1776@@ -293,3 +294,30 @@
1777 [interfaces.extend(i.complete_contexts())
1778 for i in six.itervalues(self.templates)]
1779 return interfaces
1780+
1781+ def get_incomplete_context_data(self, interfaces):
1782+ '''
1783+ Return dictionary of relation status of interfaces and any missing
1784+ required context data. Example:
1785+ {'amqp': {'missing_data': ['rabbitmq_password'], 'related': True},
1786+ 'zeromq-configuration': {'related': False}}
1787+ '''
1788+ incomplete_context_data = {}
1789+
1790+ for i in six.itervalues(self.templates):
1791+ for context in i.contexts:
1792+ for interface in interfaces:
1793+ related = False
1794+ if interface in context.interfaces:
1795+ related = context.get_related()
1796+ missing_data = context.missing_data
1797+ if missing_data:
1798+ incomplete_context_data[interface] = {'missing_data': missing_data}
1799+ if related:
1800+ if incomplete_context_data.get(interface):
1801+ incomplete_context_data[interface].update({'related': True})
1802+ else:
1803+ incomplete_context_data[interface] = {'related': True}
1804+ else:
1805+ incomplete_context_data[interface] = {'related': False}
1806+ return incomplete_context_data
1807
1808=== modified file 'hooks/charmhelpers/contrib/openstack/utils.py'
1809--- hooks/charmhelpers/contrib/openstack/utils.py 2015-07-22 12:10:31 +0000
1810+++ hooks/charmhelpers/contrib/openstack/utils.py 2015-11-11 19:55:10 +0000
1811@@ -1,5 +1,3 @@
1812-#!/usr/bin/python
1813-
1814 # Copyright 2014-2015 Canonical Limited.
1815 #
1816 # This file is part of charm-helpers.
1817@@ -24,8 +22,11 @@
1818 import json
1819 import os
1820 import sys
1821+import re
1822
1823 import six
1824+import traceback
1825+import uuid
1826 import yaml
1827
1828 from charmhelpers.contrib.network import ip
1829@@ -35,12 +36,17 @@
1830 )
1831
1832 from charmhelpers.core.hookenv import (
1833+ action_fail,
1834+ action_set,
1835 config,
1836 log as juju_log,
1837 charm_dir,
1838 INFO,
1839+ related_units,
1840 relation_ids,
1841- relation_set
1842+ relation_set,
1843+ status_set,
1844+ hook_name
1845 )
1846
1847 from charmhelpers.contrib.storage.linux.lvm import (
1848@@ -50,7 +56,8 @@
1849 )
1850
1851 from charmhelpers.contrib.network.ip import (
1852- get_ipv6_addr
1853+ get_ipv6_addr,
1854+ is_ipv6,
1855 )
1856
1857 from charmhelpers.contrib.python.packages import (
1858@@ -69,7 +76,6 @@
1859 DISTRO_PROPOSED = ('deb http://archive.ubuntu.com/ubuntu/ %s-proposed '
1860 'restricted main multiverse universe')
1861
1862-
1863 UBUNTU_OPENSTACK_RELEASE = OrderedDict([
1864 ('oneiric', 'diablo'),
1865 ('precise', 'essex'),
1866@@ -116,8 +122,41 @@
1867 ('2.2.1', 'kilo'),
1868 ('2.2.2', 'kilo'),
1869 ('2.3.0', 'liberty'),
1870+ ('2.4.0', 'liberty'),
1871+ ('2.5.0', 'liberty'),
1872 ])
1873
1874+# >= Liberty version->codename mapping
1875+PACKAGE_CODENAMES = {
1876+ 'nova-common': OrderedDict([
1877+ ('12.0.0', 'liberty'),
1878+ ]),
1879+ 'neutron-common': OrderedDict([
1880+ ('7.0.0', 'liberty'),
1881+ ]),
1882+ 'cinder-common': OrderedDict([
1883+ ('7.0.0', 'liberty'),
1884+ ]),
1885+ 'keystone': OrderedDict([
1886+ ('8.0.0', 'liberty'),
1887+ ]),
1888+ 'horizon-common': OrderedDict([
1889+ ('8.0.0', 'liberty'),
1890+ ]),
1891+ 'ceilometer-common': OrderedDict([
1892+ ('5.0.0', 'liberty'),
1893+ ]),
1894+ 'heat-common': OrderedDict([
1895+ ('5.0.0', 'liberty'),
1896+ ]),
1897+ 'glance-common': OrderedDict([
1898+ ('11.0.0', 'liberty'),
1899+ ]),
1900+ 'openstack-dashboard': OrderedDict([
1901+ ('8.0.0', 'liberty'),
1902+ ]),
1903+}
1904+
1905 DEFAULT_LOOPBACK_SIZE = '5G'
1906
1907
1908@@ -167,9 +206,9 @@
1909 error_out(e)
1910
1911
1912-def get_os_version_codename(codename):
1913+def get_os_version_codename(codename, version_map=OPENSTACK_CODENAMES):
1914 '''Determine OpenStack version number from codename.'''
1915- for k, v in six.iteritems(OPENSTACK_CODENAMES):
1916+ for k, v in six.iteritems(version_map):
1917 if v == codename:
1918 return k
1919 e = 'Could not derive OpenStack version for '\
1920@@ -201,20 +240,31 @@
1921 error_out(e)
1922
1923 vers = apt.upstream_version(pkg.current_ver.ver_str)
1924+ match = re.match('^(\d+)\.(\d+)\.(\d+)', vers)
1925+ if match:
1926+ vers = match.group(0)
1927
1928- try:
1929- if 'swift' in pkg.name:
1930- swift_vers = vers[:5]
1931- if swift_vers not in SWIFT_CODENAMES:
1932- # Deal with 1.10.0 upward
1933- swift_vers = vers[:6]
1934- return SWIFT_CODENAMES[swift_vers]
1935- else:
1936- vers = vers[:6]
1937- return OPENSTACK_CODENAMES[vers]
1938- except KeyError:
1939- e = 'Could not determine OpenStack codename for version %s' % vers
1940- error_out(e)
1941+ # >= Liberty independent project versions
1942+ if (package in PACKAGE_CODENAMES and
1943+ vers in PACKAGE_CODENAMES[package]):
1944+ return PACKAGE_CODENAMES[package][vers]
1945+ else:
1946+ # < Liberty co-ordinated project versions
1947+ try:
1948+ if 'swift' in pkg.name:
1949+ swift_vers = vers[:5]
1950+ if swift_vers not in SWIFT_CODENAMES:
1951+ # Deal with 1.10.0 upward
1952+ swift_vers = vers[:6]
1953+ return SWIFT_CODENAMES[swift_vers]
1954+ else:
1955+ vers = vers[:6]
1956+ return OPENSTACK_CODENAMES[vers]
1957+ except KeyError:
1958+ if not fatal:
1959+ return None
1960+ e = 'Could not determine OpenStack codename for version %s' % vers
1961+ error_out(e)
1962
1963
1964 def get_os_version_package(pkg, fatal=True):
1965@@ -392,7 +442,11 @@
1966 import apt_pkg as apt
1967 src = config('openstack-origin')
1968 cur_vers = get_os_version_package(package)
1969- available_vers = get_os_version_install_source(src)
1970+ if "swift" in package:
1971+ codename = get_os_codename_install_source(src)
1972+ available_vers = get_os_version_codename(codename, SWIFT_CODENAMES)
1973+ else:
1974+ available_vers = get_os_version_install_source(src)
1975 apt.init()
1976 return apt.version_compare(available_vers, cur_vers) == 1
1977
1978@@ -469,6 +523,12 @@
1979 relation_prefix=None):
1980 hosts = get_ipv6_addr(dynamic_only=False)
1981
1982+ if config('vip'):
1983+ vips = config('vip').split()
1984+ for vip in vips:
1985+ if vip and is_ipv6(vip):
1986+ hosts.append(vip)
1987+
1988 kwargs = {'database': database,
1989 'username': database_user,
1990 'hostname': json.dumps(hosts)}
1991@@ -704,3 +764,235 @@
1992 return projects[key]
1993
1994 return None
1995+
1996+
1997+def os_workload_status(configs, required_interfaces, charm_func=None):
1998+ """
1999+ Decorator to set workload status based on complete contexts
2000+ """
2001+ def wrap(f):
2002+ @wraps(f)
2003+ def wrapped_f(*args, **kwargs):
2004+ # Run the original function first
2005+ f(*args, **kwargs)
2006+ # Set workload status now that contexts have been
2007+ # acted on
2008+ set_os_workload_status(configs, required_interfaces, charm_func)
2009+ return wrapped_f
2010+ return wrap
2011+
2012+
2013+def set_os_workload_status(configs, required_interfaces, charm_func=None):
2014+ """
2015+ Set workload status based on complete contexts.
2016+ status-set missing or incomplete contexts
2017+ and juju-log details of missing required data.
2018+ charm_func is a charm specific function to run checking
2019+ for charm specific requirements such as a VIP setting.
2020+ """
2021+ incomplete_rel_data = incomplete_relation_data(configs, required_interfaces)
2022+ state = 'active'
2023+ missing_relations = []
2024+ incomplete_relations = []
2025+ message = None
2026+ charm_state = None
2027+ charm_message = None
2028+
2029+ for generic_interface in incomplete_rel_data.keys():
2030+ related_interface = None
2031+ missing_data = {}
2032+ # Related or not?
2033+ for interface in incomplete_rel_data[generic_interface]:
2034+ if incomplete_rel_data[generic_interface][interface].get('related'):
2035+ related_interface = interface
2036+ missing_data = incomplete_rel_data[generic_interface][interface].get('missing_data')
2037+ # No relation ID for the generic_interface
2038+ if not related_interface:
2039+ juju_log("{} relation is missing and must be related for "
2040+ "functionality. ".format(generic_interface), 'WARN')
2041+ state = 'blocked'
2042+ if generic_interface not in missing_relations:
2043+ missing_relations.append(generic_interface)
2044+ else:
2045+ # Relation ID exists but no related unit
2046+ if not missing_data:
2047+ # Edge case relation ID exists but departing
2048+ if ('departed' in hook_name() or 'broken' in hook_name()) \
2049+ and related_interface in hook_name():
2050+ state = 'blocked'
2051+ if generic_interface not in missing_relations:
2052+ missing_relations.append(generic_interface)
2053+ juju_log("{} relation's interface, {}, "
2054+ "relationship is departed or broken "
2055+ "and is required for functionality."
2056+ "".format(generic_interface, related_interface), "WARN")
2057+ # Normal case relation ID exists but no related unit
2058+ # (joining)
2059+ else:
2060+ juju_log("{} relations's interface, {}, is related but has "
2061+ "no units in the relation."
2062+ "".format(generic_interface, related_interface), "INFO")
2063+ # Related unit exists and data missing on the relation
2064+ else:
2065+ juju_log("{} relation's interface, {}, is related awaiting "
2066+ "the following data from the relationship: {}. "
2067+ "".format(generic_interface, related_interface,
2068+ ", ".join(missing_data)), "INFO")
2069+ if state != 'blocked':
2070+ state = 'waiting'
2071+ if generic_interface not in incomplete_relations \
2072+ and generic_interface not in missing_relations:
2073+ incomplete_relations.append(generic_interface)
2074+
2075+ if missing_relations:
2076+ message = "Missing relations: {}".format(", ".join(missing_relations))
2077+ if incomplete_relations:
2078+ message += "; incomplete relations: {}" \
2079+ "".format(", ".join(incomplete_relations))
2080+ state = 'blocked'
2081+ elif incomplete_relations:
2082+ message = "Incomplete relations: {}" \
2083+ "".format(", ".join(incomplete_relations))
2084+ state = 'waiting'
2085+
2086+ # Run charm specific checks
2087+ if charm_func:
2088+ charm_state, charm_message = charm_func(configs)
2089+ if charm_state != 'active' and charm_state != 'unknown':
2090+ state = workload_state_compare(state, charm_state)
2091+ if message:
2092+ charm_message = charm_message.replace("Incomplete relations: ",
2093+ "")
2094+ message = "{}, {}".format(message, charm_message)
2095+ else:
2096+ message = charm_message
2097+
2098+ # Set to active if all requirements have been met
2099+ if state == 'active':
2100+ message = "Unit is ready"
2101+ juju_log(message, "INFO")
2102+
2103+ status_set(state, message)
2104+
2105+
2106+def workload_state_compare(current_workload_state, workload_state):
2107+ """ Return highest priority of two states"""
2108+ hierarchy = {'unknown': -1,
2109+ 'active': 0,
2110+ 'maintenance': 1,
2111+ 'waiting': 2,
2112+ 'blocked': 3,
2113+ }
2114+
2115+ if hierarchy.get(workload_state) is None:
2116+ workload_state = 'unknown'
2117+ if hierarchy.get(current_workload_state) is None:
2118+ current_workload_state = 'unknown'
2119+
2120+ # Set workload_state based on hierarchy of statuses
2121+ if hierarchy.get(current_workload_state) > hierarchy.get(workload_state):
2122+ return current_workload_state
2123+ else:
2124+ return workload_state
2125+
2126+
2127+def incomplete_relation_data(configs, required_interfaces):
2128+ """
2129+ Check complete contexts against required_interfaces
2130+ Return dictionary of incomplete relation data.
2131+
2132+ configs is an OSConfigRenderer object with configs registered
2133+
2134+ required_interfaces is a dictionary of required general interfaces
2135+ with dictionary values of possible specific interfaces.
2136+ Example:
2137+ required_interfaces = {'database': ['shared-db', 'pgsql-db']}
2138+
2139+ The interface is said to be satisfied if anyone of the interfaces in the
2140+ list has a complete context.
2141+
2142+ Return dictionary of incomplete or missing required contexts with relation
2143+ status of interfaces and any missing data points. Example:
2144+ {'message':
2145+ {'amqp': {'missing_data': ['rabbitmq_password'], 'related': True},
2146+ 'zeromq-configuration': {'related': False}},
2147+ 'identity':
2148+ {'identity-service': {'related': False}},
2149+ 'database':
2150+ {'pgsql-db': {'related': False},
2151+ 'shared-db': {'related': True}}}
2152+ """
2153+ complete_ctxts = configs.complete_contexts()
2154+ incomplete_relations = []
2155+ for svc_type in required_interfaces.keys():
2156+ # Avoid duplicates
2157+ found_ctxt = False
2158+ for interface in required_interfaces[svc_type]:
2159+ if interface in complete_ctxts:
2160+ found_ctxt = True
2161+ if not found_ctxt:
2162+ incomplete_relations.append(svc_type)
2163+ incomplete_context_data = {}
2164+ for i in incomplete_relations:
2165+ incomplete_context_data[i] = configs.get_incomplete_context_data(required_interfaces[i])
2166+ return incomplete_context_data
2167+
2168+
2169+def do_action_openstack_upgrade(package, upgrade_callback, configs):
2170+ """Perform action-managed OpenStack upgrade.
2171+
2172+ Upgrades packages to the configured openstack-origin version and sets
2173+ the corresponding action status as a result.
2174+
2175+ If the charm was installed from source we cannot upgrade it.
2176+ For backwards compatibility a config flag (action-managed-upgrade) must
2177+ be set for this code to run, otherwise a full service level upgrade will
2178+ fire on config-changed.
2179+
2180+ @param package: package name for determining if upgrade available
2181+ @param upgrade_callback: function callback to charm's upgrade function
2182+ @param configs: templating object derived from OSConfigRenderer class
2183+
2184+ @return: True if upgrade successful; False if upgrade failed or skipped
2185+ """
2186+ ret = False
2187+
2188+ if git_install_requested():
2189+ action_set({'outcome': 'installed from source, skipped upgrade.'})
2190+ else:
2191+ if openstack_upgrade_available(package):
2192+ if config('action-managed-upgrade'):
2193+ juju_log('Upgrading OpenStack release')
2194+
2195+ try:
2196+ upgrade_callback(configs=configs)
2197+ action_set({'outcome': 'success, upgrade completed.'})
2198+ ret = True
2199+ except:
2200+ action_set({'outcome': 'upgrade failed, see traceback.'})
2201+ action_set({'traceback': traceback.format_exc()})
2202+ action_fail('do_openstack_upgrade resulted in an '
2203+ 'unexpected error')
2204+ else:
2205+ action_set({'outcome': 'action-managed-upgrade config is '
2206+ 'False, skipped upgrade.'})
2207+ else:
2208+ action_set({'outcome': 'no upgrade available.'})
2209+
2210+ return ret
2211+
2212+
2213+def remote_restart(rel_name, remote_service=None):
2214+ trigger = {
2215+ 'restart-trigger': str(uuid.uuid4()),
2216+ }
2217+ if remote_service:
2218+ trigger['remote-service'] = remote_service
2219+ for rid in relation_ids(rel_name):
2220+ # This subordinate can be related to two seperate services using
2221+ # different subordinate relations so only issue the restart if
2222+ # the principle is conencted down the relation we think it is
2223+ if related_units(relid=rid):
2224+ relation_set(relation_id=rid,
2225+ relation_settings=trigger,
2226+ )
2227
2228=== added directory 'hooks/charmhelpers/contrib/python'
2229=== added file 'hooks/charmhelpers/contrib/python/__init__.py'
2230--- hooks/charmhelpers/contrib/python/__init__.py 1970-01-01 00:00:00 +0000
2231+++ hooks/charmhelpers/contrib/python/__init__.py 2015-11-11 19:55:10 +0000
2232@@ -0,0 +1,15 @@
2233+# Copyright 2014-2015 Canonical Limited.
2234+#
2235+# This file is part of charm-helpers.
2236+#
2237+# charm-helpers is free software: you can redistribute it and/or modify
2238+# it under the terms of the GNU Lesser General Public License version 3 as
2239+# published by the Free Software Foundation.
2240+#
2241+# charm-helpers is distributed in the hope that it will be useful,
2242+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2243+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2244+# GNU Lesser General Public License for more details.
2245+#
2246+# You should have received a copy of the GNU Lesser General Public License
2247+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2248
2249=== added file 'hooks/charmhelpers/contrib/python/packages.py'
2250--- hooks/charmhelpers/contrib/python/packages.py 1970-01-01 00:00:00 +0000
2251+++ hooks/charmhelpers/contrib/python/packages.py 2015-11-11 19:55:10 +0000
2252@@ -0,0 +1,121 @@
2253+#!/usr/bin/env python
2254+# coding: utf-8
2255+
2256+# Copyright 2014-2015 Canonical Limited.
2257+#
2258+# This file is part of charm-helpers.
2259+#
2260+# charm-helpers is free software: you can redistribute it and/or modify
2261+# it under the terms of the GNU Lesser General Public License version 3 as
2262+# published by the Free Software Foundation.
2263+#
2264+# charm-helpers is distributed in the hope that it will be useful,
2265+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2266+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2267+# GNU Lesser General Public License for more details.
2268+#
2269+# You should have received a copy of the GNU Lesser General Public License
2270+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2271+
2272+import os
2273+import subprocess
2274+
2275+from charmhelpers.fetch import apt_install, apt_update
2276+from charmhelpers.core.hookenv import charm_dir, log
2277+
2278+try:
2279+ from pip import main as pip_execute
2280+except ImportError:
2281+ apt_update()
2282+ apt_install('python-pip')
2283+ from pip import main as pip_execute
2284+
2285+__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
2286+
2287+
2288+def parse_options(given, available):
2289+ """Given a set of options, check if available"""
2290+ for key, value in sorted(given.items()):
2291+ if not value:
2292+ continue
2293+ if key in available:
2294+ yield "--{0}={1}".format(key, value)
2295+
2296+
2297+def pip_install_requirements(requirements, **options):
2298+ """Install a requirements file """
2299+ command = ["install"]
2300+
2301+ available_options = ('proxy', 'src', 'log', )
2302+ for option in parse_options(options, available_options):
2303+ command.append(option)
2304+
2305+ command.append("-r {0}".format(requirements))
2306+ log("Installing from file: {} with options: {}".format(requirements,
2307+ command))
2308+ pip_execute(command)
2309+
2310+
2311+def pip_install(package, fatal=False, upgrade=False, venv=None, **options):
2312+ """Install a python package"""
2313+ if venv:
2314+ venv_python = os.path.join(venv, 'bin/pip')
2315+ command = [venv_python, "install"]
2316+ else:
2317+ command = ["install"]
2318+
2319+ available_options = ('proxy', 'src', 'log', 'index-url', )
2320+ for option in parse_options(options, available_options):
2321+ command.append(option)
2322+
2323+ if upgrade:
2324+ command.append('--upgrade')
2325+
2326+ if isinstance(package, list):
2327+ command.extend(package)
2328+ else:
2329+ command.append(package)
2330+
2331+ log("Installing {} package with options: {}".format(package,
2332+ command))
2333+ if venv:
2334+ subprocess.check_call(command)
2335+ else:
2336+ pip_execute(command)
2337+
2338+
2339+def pip_uninstall(package, **options):
2340+ """Uninstall a python package"""
2341+ command = ["uninstall", "-q", "-y"]
2342+
2343+ available_options = ('proxy', 'log', )
2344+ for option in parse_options(options, available_options):
2345+ command.append(option)
2346+
2347+ if isinstance(package, list):
2348+ command.extend(package)
2349+ else:
2350+ command.append(package)
2351+
2352+ log("Uninstalling {} package with options: {}".format(package,
2353+ command))
2354+ pip_execute(command)
2355+
2356+
2357+def pip_list():
2358+ """Returns the list of current python installed packages
2359+ """
2360+ return pip_execute(["list"])
2361+
2362+
2363+def pip_create_virtualenv(path=None):
2364+ """Create an isolated Python environment."""
2365+ apt_install('python-virtualenv')
2366+
2367+ if path:
2368+ venv_path = path
2369+ else:
2370+ venv_path = os.path.join(charm_dir(), 'venv')
2371+
2372+ if not os.path.exists(venv_path):
2373+ subprocess.check_call(['virtualenv', venv_path])
2374
2375=== modified file 'hooks/charmhelpers/contrib/storage/linux/ceph.py'
2376--- hooks/charmhelpers/contrib/storage/linux/ceph.py 2015-07-22 12:10:31 +0000
2377+++ hooks/charmhelpers/contrib/storage/linux/ceph.py 2015-11-11 19:55:10 +0000
2378@@ -28,6 +28,7 @@
2379 import shutil
2380 import json
2381 import time
2382+import uuid
2383
2384 from subprocess import (
2385 check_call,
2386@@ -35,8 +36,10 @@
2387 CalledProcessError,
2388 )
2389 from charmhelpers.core.hookenv import (
2390+ local_unit,
2391 relation_get,
2392 relation_ids,
2393+ relation_set,
2394 related_units,
2395 log,
2396 DEBUG,
2397@@ -56,6 +59,8 @@
2398 apt_install,
2399 )
2400
2401+from charmhelpers.core.kernel import modprobe
2402+
2403 KEYRING = '/etc/ceph/ceph.client.{}.keyring'
2404 KEYFILE = '/etc/ceph/ceph.client.{}.key'
2405
2406@@ -288,17 +293,6 @@
2407 os.chown(data_src_dst, uid, gid)
2408
2409
2410-# TODO: re-use
2411-def modprobe(module):
2412- """Load a kernel module and configure for auto-load on reboot."""
2413- log('Loading kernel module', level=INFO)
2414- cmd = ['modprobe', module]
2415- check_call(cmd)
2416- with open('/etc/modules', 'r+') as modules:
2417- if module not in modules.read():
2418- modules.write(module)
2419-
2420-
2421 def copy_files(src, dst, symlinks=False, ignore=None):
2422 """Copy files from src to dst."""
2423 for item in os.listdir(src):
2424@@ -411,17 +405,52 @@
2425
2426 The API is versioned and defaults to version 1.
2427 """
2428- def __init__(self, api_version=1):
2429+ def __init__(self, api_version=1, request_id=None):
2430 self.api_version = api_version
2431+ if request_id:
2432+ self.request_id = request_id
2433+ else:
2434+ self.request_id = str(uuid.uuid1())
2435 self.ops = []
2436
2437 def add_op_create_pool(self, name, replica_count=3):
2438 self.ops.append({'op': 'create-pool', 'name': name,
2439 'replicas': replica_count})
2440
2441+ def set_ops(self, ops):
2442+ """Set request ops to provided value.
2443+
2444+ Useful for injecting ops that come from a previous request
2445+ to allow comparisons to ensure validity.
2446+ """
2447+ self.ops = ops
2448+
2449 @property
2450 def request(self):
2451- return json.dumps({'api-version': self.api_version, 'ops': self.ops})
2452+ return json.dumps({'api-version': self.api_version, 'ops': self.ops,
2453+ 'request-id': self.request_id})
2454+
2455+ def _ops_equal(self, other):
2456+ if len(self.ops) == len(other.ops):
2457+ for req_no in range(0, len(self.ops)):
2458+ for key in ['replicas', 'name', 'op']:
2459+ if self.ops[req_no][key] != other.ops[req_no][key]:
2460+ return False
2461+ else:
2462+ return False
2463+ return True
2464+
2465+ def __eq__(self, other):
2466+ if not isinstance(other, self.__class__):
2467+ return False
2468+ if self.api_version == other.api_version and \
2469+ self._ops_equal(other):
2470+ return True
2471+ else:
2472+ return False
2473+
2474+ def __ne__(self, other):
2475+ return not self.__eq__(other)
2476
2477
2478 class CephBrokerRsp(object):
2479@@ -431,14 +460,198 @@
2480
2481 The API is versioned and defaults to version 1.
2482 """
2483+
2484 def __init__(self, encoded_rsp):
2485 self.api_version = None
2486 self.rsp = json.loads(encoded_rsp)
2487
2488 @property
2489+ def request_id(self):
2490+ return self.rsp.get('request-id')
2491+
2492+ @property
2493 def exit_code(self):
2494 return self.rsp.get('exit-code')
2495
2496 @property
2497 def exit_msg(self):
2498 return self.rsp.get('stderr')
2499+
2500+
2501+# Ceph Broker Conversation:
2502+# If a charm needs an action to be taken by ceph it can create a CephBrokerRq
2503+# and send that request to ceph via the ceph relation. The CephBrokerRq has a
2504+# unique id so that the client can identity which CephBrokerRsp is associated
2505+# with the request. Ceph will also respond to each client unit individually
2506+# creating a response key per client unit eg glance/0 will get a CephBrokerRsp
2507+# via key broker-rsp-glance-0
2508+#
2509+# To use this the charm can just do something like:
2510+#
2511+# from charmhelpers.contrib.storage.linux.ceph import (
2512+# send_request_if_needed,
2513+# is_request_complete,
2514+# CephBrokerRq,
2515+# )
2516+#
2517+# @hooks.hook('ceph-relation-changed')
2518+# def ceph_changed():
2519+# rq = CephBrokerRq()
2520+# rq.add_op_create_pool(name='poolname', replica_count=3)
2521+#
2522+# if is_request_complete(rq):
2523+# <Request complete actions>
2524+# else:
2525+# send_request_if_needed(get_ceph_request())
2526+#
2527+# CephBrokerRq and CephBrokerRsp are serialized into JSON. Below is an example
2528+# of glance having sent a request to ceph which ceph has successfully processed
2529+# 'ceph:8': {
2530+# 'ceph/0': {
2531+# 'auth': 'cephx',
2532+# 'broker-rsp-glance-0': '{"request-id": "0bc7dc54", "exit-code": 0}',
2533+# 'broker_rsp': '{"request-id": "0da543b8", "exit-code": 0}',
2534+# 'ceph-public-address': '10.5.44.103',
2535+# 'key': 'AQCLDttVuHXINhAAvI144CB09dYchhHyTUY9BQ==',
2536+# 'private-address': '10.5.44.103',
2537+# },
2538+# 'glance/0': {
2539+# 'broker_req': ('{"api-version": 1, "request-id": "0bc7dc54", '
2540+# '"ops": [{"replicas": 3, "name": "glance", '
2541+# '"op": "create-pool"}]}'),
2542+# 'private-address': '10.5.44.109',
2543+# },
2544+# }
2545+
2546+def get_previous_request(rid):
2547+ """Return the last ceph broker request sent on a given relation
2548+
2549+ @param rid: Relation id to query for request
2550+ """
2551+ request = None
2552+ broker_req = relation_get(attribute='broker_req', rid=rid,
2553+ unit=local_unit())
2554+ if broker_req:
2555+ request_data = json.loads(broker_req)
2556+ request = CephBrokerRq(api_version=request_data['api-version'],
2557+ request_id=request_data['request-id'])
2558+ request.set_ops(request_data['ops'])
2559+
2560+ return request
2561+
2562+
2563+def get_request_states(request):
2564+ """Return a dict of requests per relation id with their corresponding
2565+ completion state.
2566+
2567+ This allows a charm, which has a request for ceph, to see whether there is
2568+ an equivalent request already being processed and if so what state that
2569+ request is in.
2570+
2571+ @param request: A CephBrokerRq object
2572+ """
2573+ complete = []
2574+ requests = {}
2575+ for rid in relation_ids('ceph'):
2576+ complete = False
2577+ previous_request = get_previous_request(rid)
2578+ if request == previous_request:
2579+ sent = True
2580+ complete = is_request_complete_for_rid(previous_request, rid)
2581+ else:
2582+ sent = False
2583+ complete = False
2584+
2585+ requests[rid] = {
2586+ 'sent': sent,
2587+ 'complete': complete,
2588+ }
2589+
2590+ return requests
2591+
2592+
2593+def is_request_sent(request):
2594+ """Check to see if a functionally equivalent request has already been sent
2595+
2596+ Returns True if a similair request has been sent
2597+
2598+ @param request: A CephBrokerRq object
2599+ """
2600+ states = get_request_states(request)
2601+ for rid in states.keys():
2602+ if not states[rid]['sent']:
2603+ return False
2604+
2605+ return True
2606+
2607+
2608+def is_request_complete(request):
2609+ """Check to see if a functionally equivalent request has already been
2610+ completed
2611+
2612+ Returns True if a similair request has been completed
2613+
2614+ @param request: A CephBrokerRq object
2615+ """
2616+ states = get_request_states(request)
2617+ for rid in states.keys():
2618+ if not states[rid]['complete']:
2619+ return False
2620+
2621+ return True
2622+
2623+
2624+def is_request_complete_for_rid(request, rid):
2625+ """Check if a given request has been completed on the given relation
2626+
2627+ @param request: A CephBrokerRq object
2628+ @param rid: Relation ID
2629+ """
2630+ broker_key = get_broker_rsp_key()
2631+ for unit in related_units(rid):
2632+ rdata = relation_get(rid=rid, unit=unit)
2633+ if rdata.get(broker_key):
2634+ rsp = CephBrokerRsp(rdata.get(broker_key))
2635+ if rsp.request_id == request.request_id:
2636+ if not rsp.exit_code:
2637+ return True
2638+ else:
2639+ # The remote unit sent no reply targeted at this unit so either the
2640+ # remote ceph cluster does not support unit targeted replies or it
2641+ # has not processed our request yet.
2642+ if rdata.get('broker_rsp'):
2643+ request_data = json.loads(rdata['broker_rsp'])
2644+ if request_data.get('request-id'):
2645+ log('Ignoring legacy broker_rsp without unit key as remote '
2646+ 'service supports unit specific replies', level=DEBUG)
2647+ else:
2648+ log('Using legacy broker_rsp as remote service does not '
2649+ 'supports unit specific replies', level=DEBUG)
2650+ rsp = CephBrokerRsp(rdata['broker_rsp'])
2651+ if not rsp.exit_code:
2652+ return True
2653+
2654+ return False
2655+
2656+
2657+def get_broker_rsp_key():
2658+ """Return broker response key for this unit
2659+
2660+ This is the key that ceph is going to use to pass request status
2661+ information back to this unit
2662+ """
2663+ return 'broker-rsp-' + local_unit().replace('/', '-')
2664+
2665+
2666+def send_request_if_needed(request):
2667+ """Send broker request if an equivalent request has not already been sent
2668+
2669+ @param request: A CephBrokerRq object
2670+ """
2671+ if is_request_sent(request):
2672+ log('Request already sent but not complete, not sending new request',
2673+ level=DEBUG)
2674+ else:
2675+ for rid in relation_ids('ceph'):
2676+ log('Sending request {}'.format(request.request_id), level=DEBUG)
2677+ relation_set(relation_id=rid, broker_req=request.request)
2678
2679=== modified file 'hooks/charmhelpers/contrib/storage/linux/utils.py'
2680--- hooks/charmhelpers/contrib/storage/linux/utils.py 2015-02-19 22:08:13 +0000
2681+++ hooks/charmhelpers/contrib/storage/linux/utils.py 2015-11-11 19:55:10 +0000
2682@@ -43,9 +43,10 @@
2683
2684 :param block_device: str: Full path of block device to clean.
2685 '''
2686+ # https://github.com/ceph/ceph/commit/fdd7f8d83afa25c4e09aaedd90ab93f3b64a677b
2687 # sometimes sgdisk exits non-zero; this is OK, dd will clean up
2688- call(['sgdisk', '--zap-all', '--mbrtogpt',
2689- '--clear', block_device])
2690+ call(['sgdisk', '--zap-all', '--', block_device])
2691+ call(['sgdisk', '--clear', '--mbrtogpt', '--', block_device])
2692 dev_end = check_output(['blockdev', '--getsz',
2693 block_device]).decode('UTF-8')
2694 gpt_end = int(dev_end.split()[0]) - 100
2695@@ -67,4 +68,4 @@
2696 out = check_output(['mount']).decode('UTF-8')
2697 if is_partition:
2698 return bool(re.search(device + r"\b", out))
2699- return bool(re.search(device + r"[0-9]+\b", out))
2700+ return bool(re.search(device + r"[0-9]*\b", out))
2701
2702=== added file 'hooks/charmhelpers/core/files.py'
2703--- hooks/charmhelpers/core/files.py 1970-01-01 00:00:00 +0000
2704+++ hooks/charmhelpers/core/files.py 2015-11-11 19:55:10 +0000
2705@@ -0,0 +1,45 @@
2706+#!/usr/bin/env python
2707+# -*- coding: utf-8 -*-
2708+
2709+# Copyright 2014-2015 Canonical Limited.
2710+#
2711+# This file is part of charm-helpers.
2712+#
2713+# charm-helpers is free software: you can redistribute it and/or modify
2714+# it under the terms of the GNU Lesser General Public License version 3 as
2715+# published by the Free Software Foundation.
2716+#
2717+# charm-helpers is distributed in the hope that it will be useful,
2718+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2719+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2720+# GNU Lesser General Public License for more details.
2721+#
2722+# You should have received a copy of the GNU Lesser General Public License
2723+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
2724+
2725+__author__ = 'Jorge Niedbalski <niedbalski@ubuntu.com>'
2726+
2727+import os
2728+import subprocess
2729+
2730+
2731+def sed(filename, before, after, flags='g'):
2732+ """
2733+ Search and replaces the given pattern on filename.
2734+
2735+ :param filename: relative or absolute file path.
2736+ :param before: expression to be replaced (see 'man sed')
2737+ :param after: expression to replace with (see 'man sed')
2738+ :param flags: sed-compatible regex flags in example, to make
2739+ the search and replace case insensitive, specify ``flags="i"``.
2740+ The ``g`` flag is always specified regardless, so you do not
2741+ need to remember to include it when overriding this parameter.
2742+ :returns: If the sed command exit code was zero then return,
2743+ otherwise raise CalledProcessError.
2744+ """
2745+ expression = r's/{0}/{1}/{2}'.format(before,
2746+ after, flags)
2747+
2748+ return subprocess.check_call(["sed", "-i", "-r", "-e",
2749+ expression,
2750+ os.path.expanduser(filename)])
2751
2752=== modified file 'hooks/charmhelpers/core/hookenv.py'
2753--- hooks/charmhelpers/core/hookenv.py 2015-07-22 12:10:31 +0000
2754+++ hooks/charmhelpers/core/hookenv.py 2015-11-11 19:55:10 +0000
2755@@ -21,6 +21,7 @@
2756 # Charm Helpers Developers <juju@lists.ubuntu.com>
2757
2758 from __future__ import print_function
2759+import copy
2760 from distutils.version import LooseVersion
2761 from functools import wraps
2762 import glob
2763@@ -73,6 +74,7 @@
2764 res = func(*args, **kwargs)
2765 cache[key] = res
2766 return res
2767+ wrapper._wrapped = func
2768 return wrapper
2769
2770
2771@@ -172,9 +174,19 @@
2772 return os.environ.get('JUJU_RELATION', None)
2773
2774
2775-def relation_id():
2776- """The relation ID for the current relation hook"""
2777- return os.environ.get('JUJU_RELATION_ID', None)
2778+@cached
2779+def relation_id(relation_name=None, service_or_unit=None):
2780+ """The relation ID for the current or a specified relation"""
2781+ if not relation_name and not service_or_unit:
2782+ return os.environ.get('JUJU_RELATION_ID', None)
2783+ elif relation_name and service_or_unit:
2784+ service_name = service_or_unit.split('/')[0]
2785+ for relid in relation_ids(relation_name):
2786+ remote_service = remote_service_name(relid)
2787+ if remote_service == service_name:
2788+ return relid
2789+ else:
2790+ raise ValueError('Must specify neither or both of relation_name and service_or_unit')
2791
2792
2793 def local_unit():
2794@@ -192,9 +204,20 @@
2795 return local_unit().split('/')[0]
2796
2797
2798+@cached
2799+def remote_service_name(relid=None):
2800+ """The remote service name for a given relation-id (or the current relation)"""
2801+ if relid is None:
2802+ unit = remote_unit()
2803+ else:
2804+ units = related_units(relid)
2805+ unit = units[0] if units else None
2806+ return unit.split('/')[0] if unit else None
2807+
2808+
2809 def hook_name():
2810 """The name of the currently executing hook"""
2811- return os.path.basename(sys.argv[0])
2812+ return os.environ.get('JUJU_HOOK_NAME', os.path.basename(sys.argv[0]))
2813
2814
2815 class Config(dict):
2816@@ -263,7 +286,7 @@
2817 self.path = path or self.path
2818 with open(self.path) as f:
2819 self._prev_dict = json.load(f)
2820- for k, v in self._prev_dict.items():
2821+ for k, v in copy.deepcopy(self._prev_dict).items():
2822 if k not in self:
2823 self[k] = v
2824
2825@@ -468,6 +491,76 @@
2826
2827
2828 @cached
2829+def peer_relation_id():
2830+ '''Get a peer relation id if a peer relation has been joined, else None.'''
2831+ md = metadata()
2832+ section = md.get('peers')
2833+ if section:
2834+ for key in section:
2835+ relids = relation_ids(key)
2836+ if relids:
2837+ return relids[0]
2838+ return None
2839+
2840+
2841+@cached
2842+def relation_to_interface(relation_name):
2843+ """
2844+ Given the name of a relation, return the interface that relation uses.
2845+
2846+ :returns: The interface name, or ``None``.
2847+ """
2848+ return relation_to_role_and_interface(relation_name)[1]
2849+
2850+
2851+@cached
2852+def relation_to_role_and_interface(relation_name):
2853+ """
2854+ Given the name of a relation, return the role and the name of the interface
2855+ that relation uses (where role is one of ``provides``, ``requires``, or ``peer``).
2856+
2857+ :returns: A tuple containing ``(role, interface)``, or ``(None, None)``.
2858+ """
2859+ _metadata = metadata()
2860+ for role in ('provides', 'requires', 'peer'):
2861+ interface = _metadata.get(role, {}).get(relation_name, {}).get('interface')
2862+ if interface:
2863+ return role, interface
2864+ return None, None
2865+
2866+
2867+@cached
2868+def role_and_interface_to_relations(role, interface_name):
2869+ """
2870+ Given a role and interface name, return a list of relation names for the
2871+ current charm that use that interface under that role (where role is one
2872+ of ``provides``, ``requires``, or ``peer``).
2873+
2874+ :returns: A list of relation names.
2875+ """
2876+ _metadata = metadata()
2877+ results = []
2878+ for relation_name, relation in _metadata.get(role, {}).items():
2879+ if relation['interface'] == interface_name:
2880+ results.append(relation_name)
2881+ return results
2882+
2883+
2884+@cached
2885+def interface_to_relations(interface_name):
2886+ """
2887+ Given an interface, return a list of relation names for the current
2888+ charm that use that interface.
2889+
2890+ :returns: A list of relation names.
2891+ """
2892+ results = []
2893+ for role in ('provides', 'requires', 'peer'):
2894+ results.extend(role_and_interface_to_relations(role, interface_name))
2895+ return results
2896+
2897+
2898+@cached
2899 def charm_name():
2900 """Get the name of the current charm as is specified on metadata.yaml"""
2901 return metadata().get('name')
2902@@ -543,6 +636,38 @@
2903 return unit_get('private-address')
2904
2905
2906+@cached
2907+def storage_get(attribute="", storage_id=""):
2908+ """Get storage attributes"""
2909+ _args = ['storage-get', '--format=json']
2910+ if storage_id:
2911+ _args.extend(('-s', storage_id))
2912+ if attribute:
2913+ _args.append(attribute)
2914+ try:
2915+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2916+ except ValueError:
2917+ return None
2918+
2919+
2920+@cached
2921+def storage_list(storage_name=""):
2922+ """List the storage IDs for the unit"""
2923+ _args = ['storage-list', '--format=json']
2924+ if storage_name:
2925+ _args.append(storage_name)
2926+ try:
2927+ return json.loads(subprocess.check_output(_args).decode('UTF-8'))
2928+ except ValueError:
2929+ return None
2930+ except OSError as e:
2931+ import errno
2932+ if e.errno == errno.ENOENT:
2933+ # storage-list does not exist
2934+ return []
2935+ raise
2936+
2937+
2938 class UnregisteredHookError(Exception):
2939 """Raised when an undefined hook is called"""
2940 pass
2941@@ -643,6 +768,21 @@
2942 subprocess.check_call(['action-fail', message])
2943
2944
2945+def action_name():
2946+ """Get the name of the currently executing action."""
2947+ return os.environ.get('JUJU_ACTION_NAME')
2948+
2949+
2950+def action_uuid():
2951+ """Get the UUID of the currently executing action."""
2952+ return os.environ.get('JUJU_ACTION_UUID')
2953+
2954+
2955+def action_tag():
2956+ """Get the tag for the currently executing action."""
2957+ return os.environ.get('JUJU_ACTION_TAG')
2958+
2959+
2960 def status_set(workload_state, message):
2961 """Set the workload state with a message
2962
2963@@ -672,25 +812,28 @@
2964
2965
2966 def status_get():
2967- """Retrieve the previously set juju workload state
2968-
2969- If the status-set command is not found then assume this is juju < 1.23 and
2970- return 'unknown'
2971+ """Retrieve the previously set juju workload state and message
2972+
2973+ If the status-get command is not found then assume this is juju < 1.23 and
2974+ return 'unknown', ""
2975+
2976 """
2977- cmd = ['status-get']
2978+ cmd = ['status-get', "--format=json", "--include-data"]
2979 try:
2980- raw_status = subprocess.check_output(cmd, universal_newlines=True)
2981- status = raw_status.rstrip()
2982- return status
2983+ raw_status = subprocess.check_output(cmd)
2984 except OSError as e:
2985 if e.errno == errno.ENOENT:
2986- return 'unknown'
2987+ return ('unknown', "")
2988 else:
2989 raise
2990+ else:
2991+ status = json.loads(raw_status.decode("UTF-8"))
2992+ return (status["status"], status["message"])
2993
2994
2995 def translate_exc(from_exc, to_exc):
2996 def inner_translate_exc1(f):
2997+ @wraps(f)
2998 def inner_translate_exc2(*args, **kwargs):
2999 try:
3000 return f(*args, **kwargs)
3001
3002=== modified file 'hooks/charmhelpers/core/host.py'
3003--- hooks/charmhelpers/core/host.py 2015-07-22 12:10:31 +0000
3004+++ hooks/charmhelpers/core/host.py 2015-11-11 19:55:10 +0000
3005@@ -63,32 +63,48 @@
3006 return service_result
3007
3008
3009-def service_pause(service_name, init_dir=None):
3010+def service_pause(service_name, init_dir="/etc/init", initd_dir="/etc/init.d"):
3011 """Pause a system service.
3012
3013 Stop it, and prevent it from starting again at boot."""
3014- if init_dir is None:
3015- init_dir = "/etc/init"
3016 stopped = service_stop(service_name)
3017- # XXX: Support systemd too
3018- override_path = os.path.join(
3019- init_dir, '{}.conf.override'.format(service_name))
3020- with open(override_path, 'w') as fh:
3021- fh.write("manual\n")
3022+ upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
3023+ sysv_file = os.path.join(initd_dir, service_name)
3024+ if os.path.exists(upstart_file):
3025+ override_path = os.path.join(
3026+ init_dir, '{}.override'.format(service_name))
3027+ with open(override_path, 'w') as fh:
3028+ fh.write("manual\n")
3029+ elif os.path.exists(sysv_file):
3030+ subprocess.check_call(["update-rc.d", service_name, "disable"])
3031+ else:
3032+ # XXX: Support SystemD too
3033+ raise ValueError(
3034+ "Unable to detect {0} as either Upstart {1} or SysV {2}".format(
3035+ service_name, upstart_file, sysv_file))
3036 return stopped
3037
3038
3039-def service_resume(service_name, init_dir=None):
3040+def service_resume(service_name, init_dir="/etc/init",
3041+ initd_dir="/etc/init.d"):
3042 """Resume a system service.
3043
3044 Reenable starting again at boot. Start the service"""
3045- # XXX: Support systemd too
3046- if init_dir is None:
3047- init_dir = "/etc/init"
3048- override_path = os.path.join(
3049- init_dir, '{}.conf.override'.format(service_name))
3050- if os.path.exists(override_path):
3051- os.unlink(override_path)
3052+ upstart_file = os.path.join(init_dir, "{}.conf".format(service_name))
3053+ sysv_file = os.path.join(initd_dir, service_name)
3054+ if os.path.exists(upstart_file):
3055+ override_path = os.path.join(
3056+ init_dir, '{}.override'.format(service_name))
3057+ if os.path.exists(override_path):
3058+ os.unlink(override_path)
3059+ elif os.path.exists(sysv_file):
3060+ subprocess.check_call(["update-rc.d", service_name, "enable"])
3061+ else:
3062+ # XXX: Support SystemD too
3063+ raise ValueError(
3064+ "Unable to detect {0} as either Upstart {1} or SysV {2}".format(
3065+ service_name, upstart_file, sysv_file))
3066+
3067 started = service_start(service_name)
3068 return started
3069
3070@@ -148,6 +164,16 @@
3071 return user_info
3072
3073
3074+def user_exists(username):
3075+ """Check if a user exists"""
3076+ try:
3077+ pwd.getpwnam(username)
3078+ user_exists = True
3079+ except KeyError:
3080+ user_exists = False
3081+ return user_exists
3082+
3083+
3084 def add_group(group_name, system_group=False):
3085 """Add a group to the system"""
3086 try:
3087@@ -280,6 +306,17 @@
3088 return system_mounts
3089
3090
3091+def fstab_mount(mountpoint):
3092+ """Mount filesystem using fstab"""
3093+ cmd_args = ['mount', mountpoint]
3094+ try:
3095+ subprocess.check_output(cmd_args)
3096+ except subprocess.CalledProcessError as e:
3097+ log('Error unmounting {}\n{}'.format(mountpoint, e.output))
3098+ return False
3099+ return True
3100+
3101+
3102 def file_hash(path, hash_type='md5'):
3103 """
3104 Generate a hash checksum of the contents of 'path' or None if not found.
3105@@ -396,25 +433,80 @@
3106 return(''.join(random_chars))
3107
3108
3109-def list_nics(nic_type):
3110+def is_phy_iface(interface):
3111+ """Returns True if interface is not virtual, otherwise False."""
3112+ if interface:
3113+ sys_net = '/sys/class/net'
3114+ if os.path.isdir(sys_net):
3115+ for iface in glob.glob(os.path.join(sys_net, '*')):
3116+ if '/virtual/' in os.path.realpath(iface):
3117+ continue
3118+
3119+ if interface == os.path.basename(iface):
3120+ return True
3121+
3122+ return False
3123+
3124+
3125+def get_bond_master(interface):
3126+ """Returns bond master if interface is bond slave otherwise None.
3127+
3128+ NOTE: the provided interface is expected to be physical
3129+ """
3130+ if interface:
3131+ iface_path = '/sys/class/net/%s' % (interface)
3132+ if os.path.exists(iface_path):
3133+ if '/virtual/' in os.path.realpath(iface_path):
3134+ return None
3135+
3136+ master = os.path.join(iface_path, 'master')
3137+ if os.path.exists(master):
3138+ master = os.path.realpath(master)
3139+ # make sure it is a bond master
3140+ if os.path.exists(os.path.join(master, 'bonding')):
3141+ return os.path.basename(master)
3142+
3143+ return None
3144+
3145+
3146+def list_nics(nic_type=None):
3147 '''Return a list of nics of given type(s)'''
3148 if isinstance(nic_type, six.string_types):
3149 int_types = [nic_type]
3150 else:
3151 int_types = nic_type
3152+
3153 interfaces = []
3154- for int_type in int_types:
3155- cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
3156+ if nic_type:
3157+ for int_type in int_types:
3158+ cmd = ['ip', 'addr', 'show', 'label', int_type + '*']
3159+ ip_output = subprocess.check_output(cmd).decode('UTF-8')
3160+ ip_output = ip_output.split('\n')
3161+ ip_output = (line for line in ip_output if line)
3162+ for line in ip_output:
3163+ if line.split()[1].startswith(int_type):
3164+ matched = re.search('.*: (' + int_type +
3165+ r'[0-9]+\.[0-9]+)@.*', line)
3166+ if matched:
3167+ iface = matched.groups()[0]
3168+ else:
3169+ iface = line.split()[1].replace(":", "")
3170+
3171+ if iface not in interfaces:
3172+ interfaces.append(iface)
3173+ else:
3174+ cmd = ['ip', 'a']
3175 ip_output = subprocess.check_output(cmd).decode('UTF-8').split('\n')
3176- ip_output = (line for line in ip_output if line)
3177+ ip_output = (line.strip() for line in ip_output if line)
3178+
3179+ key = re.compile('^[0-9]+:\s+(.+):')
3180 for line in ip_output:
3181- if line.split()[1].startswith(int_type):
3182- matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
3183- if matched:
3184- interface = matched.groups()[0]
3185- else:
3186- interface = line.split()[1].replace(":", "")
3187- interfaces.append(interface)
3188+ matched = re.search(key, line)
3189+ if matched:
3190+ iface = matched.group(1)
3191+ iface = iface.partition("@")[0]
3192+ if iface not in interfaces:
3193+ interfaces.append(iface)
3194
3195 return interfaces
3196
3197@@ -474,7 +566,14 @@
3198 os.chdir(cur)
3199
3200
3201-def chownr(path, owner, group, follow_links=True):
3202+def chownr(path, owner, group, follow_links=True, chowntopdir=False):
3203+ """
3204+ Recursively change user and group ownership of files and directories
3205+ in given path. Doesn't chown path itself by default, only its children.
3206+
3207+ :param bool follow_links: Also Chown links if True
3208+ :param bool chowntopdir: Also chown path itself if True
3209+ """
3210 uid = pwd.getpwnam(owner).pw_uid
3211 gid = grp.getgrnam(group).gr_gid
3212 if follow_links:
3213@@ -482,6 +581,10 @@
3214 else:
3215 chown = os.lchown
3216
3217+ if chowntopdir:
3218+ broken_symlink = os.path.lexists(path) and not os.path.exists(path)
3219+ if not broken_symlink:
3220+ chown(path, uid, gid)
3221 for root, dirs, files in os.walk(path):
3222 for name in dirs + files:
3223 full = os.path.join(root, name)
3224@@ -492,3 +595,19 @@
3225
3226 def lchownr(path, owner, group):
3227 chownr(path, owner, group, follow_links=False)
3228+
3229+
3230+def get_total_ram():
3231+ '''The total amount of system RAM in bytes.
3232+
3233+ This is what is reported by the OS, and may be overcommitted when
3234+ there are multiple containers hosted on the same machine.
3235+ '''
3236+ with open('/proc/meminfo', 'r') as f:
3237+ for line in f.readlines():
3238+ if line:
3239+ key, value, unit = line.split()
3240+ if key == 'MemTotal:':
3241+ assert unit == 'kB', 'Unknown unit'
3242+ return int(value) * 1024 # Classic, not KiB.
3243+ raise NotImplementedError()
3244
3245=== added file 'hooks/charmhelpers/core/hugepage.py'
3246--- hooks/charmhelpers/core/hugepage.py 1970-01-01 00:00:00 +0000
3247+++ hooks/charmhelpers/core/hugepage.py 2015-11-11 19:55:10 +0000
3248@@ -0,0 +1,71 @@
3249+# -*- coding: utf-8 -*-
3250+
3251+# Copyright 2014-2015 Canonical Limited.
3252+#
3253+# This file is part of charm-helpers.
3254+#
3255+# charm-helpers is free software: you can redistribute it and/or modify
3256+# it under the terms of the GNU Lesser General Public License version 3 as
3257+# published by the Free Software Foundation.
3258+#
3259+# charm-helpers is distributed in the hope that it will be useful,
3260+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3261+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3262+# GNU Lesser General Public License for more details.
3263+#
3264+# You should have received a copy of the GNU Lesser General Public License
3265+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3266+
3267+import yaml
3268+from charmhelpers.core import fstab
3269+from charmhelpers.core import sysctl
3270+from charmhelpers.core.host import (
3271+ add_group,
3272+ add_user_to_group,
3273+ fstab_mount,
3274+ mkdir,
3275+)
3276+from charmhelpers.core.strutils import bytes_from_string
3277+from subprocess import check_output
3278+
3279+
3280+def hugepage_support(user, group='hugetlb', nr_hugepages=256,
3281+ max_map_count=65536, mnt_point='/run/hugepages/kvm',
3282+ pagesize='2MB', mount=True, set_shmmax=False):
3283+ """Enable hugepages on system.
3284+
3285+ Args:
3286+ user (str) -- Username to allow access to hugepages to
3287+ group (str) -- Group name to own hugepages
3288+ nr_hugepages (int) -- Number of pages to reserve
3289+ max_map_count (int) -- Number of Virtual Memory Areas a process can own
3290+ mnt_point (str) -- Directory to mount hugepages on
3291+ pagesize (str) -- Size of hugepages
3292+ mount (bool) -- Whether to Mount hugepages
3293+ """
3294+ group_info = add_group(group)
3295+ gid = group_info.gr_gid
3296+ add_user_to_group(user, group)
3297+ if max_map_count < 2 * nr_hugepages:
3298+ max_map_count = 2 * nr_hugepages
3299+ sysctl_settings = {
3300+ 'vm.nr_hugepages': nr_hugepages,
3301+ 'vm.max_map_count': max_map_count,
3302+ 'vm.hugetlb_shm_group': gid,
3303+ }
3304+ if set_shmmax:
3305+ shmmax_current = int(check_output(['sysctl', '-n', 'kernel.shmmax']))
3306+ shmmax_minsize = bytes_from_string(pagesize) * nr_hugepages
3307+ if shmmax_minsize > shmmax_current:
3308+ sysctl_settings['kernel.shmmax'] = shmmax_minsize
3309+ sysctl.create(yaml.dump(sysctl_settings), '/etc/sysctl.d/10-hugepage.conf')
3310+ mkdir(mnt_point, owner='root', group='root', perms=0o755, force=False)
3311+ lfstab = fstab.Fstab()
3312+ fstab_entry = lfstab.get_entry_by_attr('mountpoint', mnt_point)
3313+ if fstab_entry:
3314+ lfstab.remove_entry(fstab_entry)
3315+ entry = lfstab.Entry('nodev', mnt_point, 'hugetlbfs',
3316+ 'mode=1770,gid={},pagesize={}'.format(gid, pagesize), 0, 0)
3317+ lfstab.add_entry(entry)
3318+ if mount:
3319+ fstab_mount(mnt_point)
3320
3321=== added file 'hooks/charmhelpers/core/kernel.py'
3322--- hooks/charmhelpers/core/kernel.py 1970-01-01 00:00:00 +0000
3323+++ hooks/charmhelpers/core/kernel.py 2015-11-11 19:55:10 +0000
3324@@ -0,0 +1,68 @@
3325+#!/usr/bin/env python
3326+# -*- coding: utf-8 -*-
3327+
3328+# Copyright 2014-2015 Canonical Limited.
3329+#
3330+# This file is part of charm-helpers.
3331+#
3332+# charm-helpers is free software: you can redistribute it and/or modify
3333+# it under the terms of the GNU Lesser General Public License version 3 as
3334+# published by the Free Software Foundation.
3335+#
3336+# charm-helpers is distributed in the hope that it will be useful,
3337+# but WITHOUT ANY WARRANTY; without even the implied warranty of
3338+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3339+# GNU Lesser General Public License for more details.
3340+#
3341+# You should have received a copy of the GNU Lesser General Public License
3342+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3343+
3344+__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
3345+
3346+from charmhelpers.core.hookenv import (
3347+ log,
3348+ INFO
3349+)
3350+
3351+from subprocess import check_call, check_output
3352+import re
3353+
3354+
3355+def modprobe(module, persist=True):
3356+ """Load a kernel module and configure for auto-load on reboot."""
3357+ cmd = ['modprobe', module]
3358+
3359+ log('Loading kernel module %s' % module, level=INFO)
3360+
3361+ check_call(cmd)
3362+ if persist:
3363+ with open('/etc/modules', 'r+') as modules:
3364+ if module not in modules.read():
3365+ modules.write(module)
3366+
3367+
3368+def rmmod(module, force=False):
3369+ """Remove a module from the linux kernel"""
3370+ cmd = ['rmmod']
3371+ if force:
3372+ cmd.append('-f')
3373+ cmd.append(module)
3374+ log('Removing kernel module %s' % module, level=INFO)
3375+ return check_call(cmd)
3376+
3377+
3378+def lsmod():
3379+ """Shows what kernel modules are currently loaded"""
3380+ return check_output(['lsmod'],
3381+ universal_newlines=True)
3382+
3383+
3384+def is_module_loaded(module):
3385+ """Checks if a kernel module is already loaded"""
3386+ matches = re.findall('^%s[ ]+' % module, lsmod(), re.M)
3387+ return len(matches) > 0
3388+
3389+
3390+def update_initramfs(version='all'):
3391+ """Updates an initramfs image"""
3392+ return check_call(["update-initramfs", "-k", version, "-u"])
3393
3394=== modified file 'hooks/charmhelpers/core/services/helpers.py'
3395--- hooks/charmhelpers/core/services/helpers.py 2015-07-22 12:10:31 +0000
3396+++ hooks/charmhelpers/core/services/helpers.py 2015-11-11 19:55:10 +0000
3397@@ -16,7 +16,9 @@
3398
3399 import os
3400 import yaml
3401+
3402 from charmhelpers.core import hookenv
3403+from charmhelpers.core import host
3404 from charmhelpers.core import templating
3405
3406 from charmhelpers.core.services.base import ManagerCallback
3407@@ -240,27 +242,44 @@
3408
3409 :param str source: The template source file, relative to
3410 `$CHARM_DIR/templates`
3411+
3412 :param str target: The target to write the rendered template to
3413 :param str owner: The owner of the rendered file
3414 :param str group: The group of the rendered file
3415 :param int perms: The permissions of the rendered file
3416-
3417+ :param partial on_change_action: functools partial to be executed when
3418+ rendered file changes
3419+ :param jinja2 loader template_loader: A jinja2 template loader
3420 """
3421 def __init__(self, source, target,
3422- owner='root', group='root', perms=0o444):
3423+ owner='root', group='root', perms=0o444,
3424+ on_change_action=None, template_loader=None):
3425 self.source = source
3426 self.target = target
3427 self.owner = owner
3428 self.group = group
3429 self.perms = perms
3430+ self.on_change_action = on_change_action
3431+ self.template_loader = template_loader
3432
3433 def __call__(self, manager, service_name, event_name):
3434+ pre_checksum = ''
3435+ if self.on_change_action and os.path.isfile(self.target):
3436+ pre_checksum = host.file_hash(self.target)
3437 service = manager.get_service(service_name)
3438 context = {}
3439 for ctx in service.get('required_data', []):
3440 context.update(ctx)
3441 templating.render(self.source, self.target, context,
3442- self.owner, self.group, self.perms)
3443+ self.owner, self.group, self.perms,
3444+ template_loader=self.template_loader)
3445+ if self.on_change_action:
3446+ if pre_checksum == host.file_hash(self.target):
3447+ hookenv.log(
3448+ 'No change detected: {}'.format(self.target),
3449+ hookenv.DEBUG)
3450+ else:
3451+ self.on_change_action()
3452
3453
3454 # Convenience aliases for templates
3455
3456=== modified file 'hooks/charmhelpers/core/strutils.py'
3457--- hooks/charmhelpers/core/strutils.py 2015-07-22 12:10:31 +0000
3458+++ hooks/charmhelpers/core/strutils.py 2015-11-11 19:55:10 +0000
3459@@ -18,6 +18,7 @@
3460 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3461
3462 import six
3463+import re
3464
3465
3466 def bool_from_string(value):
3467@@ -40,3 +41,32 @@
3468
3469 msg = "Unable to interpret string value '%s' as boolean" % (value)
3470 raise ValueError(msg)
3471+
3472+
3473+def bytes_from_string(value):
3474+ """Interpret human readable string value as bytes.
3475+
3476+ Returns int
3477+ """
3478+ BYTE_POWER = {
3479+ 'K': 1,
3480+ 'KB': 1,
3481+ 'M': 2,
3482+ 'MB': 2,
3483+ 'G': 3,
3484+ 'GB': 3,
3485+ 'T': 4,
3486+ 'TB': 4,
3487+ 'P': 5,
3488+ 'PB': 5,
3489+ }
3490+ if isinstance(value, six.string_types):
3491+ value = six.text_type(value)
3492+ else:
3493+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
3494+ raise ValueError(msg)
3495+ matches = re.match("([0-9]+)([a-zA-Z]+)", value)
3496+ if not matches:
3497+ msg = "Unable to interpret string value '%s' as bytes" % (value)
3498+ raise ValueError(msg)
3499+ return int(matches.group(1)) * (1024 ** BYTE_POWER[matches.group(2)])
3500
3501=== modified file 'hooks/charmhelpers/core/templating.py'
3502--- hooks/charmhelpers/core/templating.py 2015-02-19 22:08:13 +0000
3503+++ hooks/charmhelpers/core/templating.py 2015-11-11 19:55:10 +0000
3504@@ -21,7 +21,7 @@
3505
3506
3507 def render(source, target, context, owner='root', group='root',
3508- perms=0o444, templates_dir=None, encoding='UTF-8'):
3509+ perms=0o444, templates_dir=None, encoding='UTF-8', template_loader=None):
3510 """
3511 Render a template.
3512
3513@@ -52,17 +52,24 @@
3514 apt_install('python-jinja2', fatal=True)
3515 from jinja2 import FileSystemLoader, Environment, exceptions
3516
3517- if templates_dir is None:
3518- templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
3519- loader = Environment(loader=FileSystemLoader(templates_dir))
3520+ if template_loader:
3521+ template_env = Environment(loader=template_loader)
3522+ else:
3523+ if templates_dir is None:
3524+ templates_dir = os.path.join(hookenv.charm_dir(), 'templates')
3525+ template_env = Environment(loader=FileSystemLoader(templates_dir))
3526 try:
3527 source = source
3528- template = loader.get_template(source)
3529+ template = template_env.get_template(source)
3530 except exceptions.TemplateNotFound as e:
3531 hookenv.log('Could not load template %s from %s.' %
3532 (source, templates_dir),
3533 level=hookenv.ERROR)
3534 raise e
3535 content = template.render(context)
3536- host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
3537+ target_dir = os.path.dirname(target)
3538+ if not os.path.exists(target_dir):
3539+ # This is a terrible default directory permission, as the file
3540+ # or its siblings will often contain secrets.
3541+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
3542 host.write_file(target, content.encode(encoding), owner, group, perms)
3543
3544=== modified file 'hooks/charmhelpers/core/unitdata.py'
3545--- hooks/charmhelpers/core/unitdata.py 2015-07-22 12:10:31 +0000
3546+++ hooks/charmhelpers/core/unitdata.py 2015-11-11 19:55:10 +0000
3547@@ -152,6 +152,7 @@
3548 import collections
3549 import contextlib
3550 import datetime
3551+import itertools
3552 import json
3553 import os
3554 import pprint
3555@@ -164,8 +165,7 @@
3556 class Storage(object):
3557 """Simple key value database for local unit state within charms.
3558
3559- Modifications are automatically committed at hook exit. That's
3560- currently regardless of exit code.
3561+ Modifications are not persisted unless :meth:`flush` is called.
3562
3563 To support dicts, lists, integer, floats, and booleans values
3564 are automatically json encoded/decoded.
3565@@ -173,8 +173,11 @@
3566 def __init__(self, path=None):
3567 self.db_path = path
3568 if path is None:
3569- self.db_path = os.path.join(
3570- os.environ.get('CHARM_DIR', ''), '.unit-state.db')
3571+ if 'UNIT_STATE_DB' in os.environ:
3572+ self.db_path = os.environ['UNIT_STATE_DB']
3573+ else:
3574+ self.db_path = os.path.join(
3575+ os.environ.get('CHARM_DIR', ''), '.unit-state.db')
3576 self.conn = sqlite3.connect('%s' % self.db_path)
3577 self.cursor = self.conn.cursor()
3578 self.revision = None
3579@@ -189,15 +192,8 @@
3580 self.conn.close()
3581 self._closed = True
3582
3583- def _scoped_query(self, stmt, params=None):
3584- if params is None:
3585- params = []
3586- return stmt, params
3587-
3588 def get(self, key, default=None, record=False):
3589- self.cursor.execute(
3590- *self._scoped_query(
3591- 'select data from kv where key=?', [key]))
3592+ self.cursor.execute('select data from kv where key=?', [key])
3593 result = self.cursor.fetchone()
3594 if not result:
3595 return default
3596@@ -206,33 +202,81 @@
3597 return json.loads(result[0])
3598
3599 def getrange(self, key_prefix, strip=False):
3600- stmt = "select key, data from kv where key like '%s%%'" % key_prefix
3601- self.cursor.execute(*self._scoped_query(stmt))
3602+ """
3603+ Get a range of keys starting with a common prefix as a mapping of
3604+ keys to values.
3605+
3606+ :param str key_prefix: Common prefix among all keys
3607+ :param bool strip: Optionally strip the common prefix from the key
3608+ names in the returned dict
3609+ :return dict: A (possibly empty) dict of key-value mappings
3610+ """
3611+ self.cursor.execute("select key, data from kv where key like ?",
3612+ ['%s%%' % key_prefix])
3613 result = self.cursor.fetchall()
3614
3615 if not result:
3616- return None
3617+ return {}
3618 if not strip:
3619 key_prefix = ''
3620 return dict([
3621 (k[len(key_prefix):], json.loads(v)) for k, v in result])
3622
3623 def update(self, mapping, prefix=""):
3624+ """
3625+ Set the values of multiple keys at once.
3626+
3627+ :param dict mapping: Mapping of keys to values
3628+ :param str prefix: Optional prefix to apply to all keys in `mapping`
3629+ before setting
3630+ """
3631 for k, v in mapping.items():
3632 self.set("%s%s" % (prefix, k), v)
3633
3634 def unset(self, key):
3635+ """
3636+ Remove a key from the database entirely.
3637+ """
3638 self.cursor.execute('delete from kv where key=?', [key])
3639 if self.revision and self.cursor.rowcount:
3640 self.cursor.execute(
3641 'insert into kv_revisions values (?, ?, ?)',
3642 [key, self.revision, json.dumps('DELETED')])
3643
3644+ def unsetrange(self, keys=None, prefix=""):
3645+ """
3646+ Remove a range of keys starting with a common prefix, from the database
3647+ entirely.
3648+
3649+ :param list keys: List of keys to remove.
3650+ :param str prefix: Optional prefix to apply to all keys in ``keys``
3651+ before removing.
3652+ """
3653+ if keys is not None:
3654+ keys = ['%s%s' % (prefix, key) for key in keys]
3655+ self.cursor.execute('delete from kv where key in (%s)' % ','.join(['?'] * len(keys)), keys)
3656+ if self.revision and self.cursor.rowcount:
3657+ self.cursor.execute(
3658+ 'insert into kv_revisions values %s' % ','.join(['(?, ?, ?)'] * len(keys)),
3659+ list(itertools.chain.from_iterable((key, self.revision, json.dumps('DELETED')) for key in keys)))
3660+ else:
3661+ self.cursor.execute('delete from kv where key like ?',
3662+ ['%s%%' % prefix])
3663+ if self.revision and self.cursor.rowcount:
3664+ self.cursor.execute(
3665+ 'insert into kv_revisions values (?, ?, ?)',
3666+ ['%s%%' % prefix, self.revision, json.dumps('DELETED')])
3667+
3668 def set(self, key, value):
3669+ """
3670+ Set a value in the database.
3671+
3672+ :param str key: Key to set the value for
3673+ :param value: Any JSON-serializable value to be set
3674+ """
3675 serialized = json.dumps(value)
3676
3677- self.cursor.execute(
3678- 'select data from kv where key=?', [key])
3679+ self.cursor.execute('select data from kv where key=?', [key])
3680 exists = self.cursor.fetchone()
3681
3682 # Skip mutations to the same value
3683
3684=== modified file 'hooks/charmhelpers/fetch/__init__.py'
3685--- hooks/charmhelpers/fetch/__init__.py 2015-07-22 12:10:31 +0000
3686+++ hooks/charmhelpers/fetch/__init__.py 2015-11-11 19:55:10 +0000
3687@@ -90,6 +90,14 @@
3688 'kilo/proposed': 'trusty-proposed/kilo',
3689 'trusty-kilo/proposed': 'trusty-proposed/kilo',
3690 'trusty-proposed/kilo': 'trusty-proposed/kilo',
3691+ # Liberty
3692+ 'liberty': 'trusty-updates/liberty',
3693+ 'trusty-liberty': 'trusty-updates/liberty',
3694+ 'trusty-liberty/updates': 'trusty-updates/liberty',
3695+ 'trusty-updates/liberty': 'trusty-updates/liberty',
3696+ 'liberty/proposed': 'trusty-proposed/liberty',
3697+ 'trusty-liberty/proposed': 'trusty-proposed/liberty',
3698+ 'trusty-proposed/liberty': 'trusty-proposed/liberty',
3699 }
3700
3701 # The order of this list is very important. Handlers should be listed in from
3702@@ -217,12 +225,12 @@
3703
3704 def apt_mark(packages, mark, fatal=False):
3705 """Flag one or more packages using apt-mark"""
3706+ log("Marking {} as {}".format(packages, mark))
3707 cmd = ['apt-mark', mark]
3708 if isinstance(packages, six.string_types):
3709 cmd.append(packages)
3710 else:
3711 cmd.extend(packages)
3712- log("Holding {}".format(packages))
3713
3714 if fatal:
3715 subprocess.check_call(cmd, universal_newlines=True)
3716
3717=== modified file 'metadata.yaml'
3718--- metadata.yaml 2015-02-25 15:27:38 +0000
3719+++ metadata.yaml 2015-11-11 19:55:10 +0000
3720@@ -6,7 +6,7 @@
3721 virtual-network to virtual-machines, containers or network namespaces.
3722 .
3723 This charm provides the controller component.
3724-categories:
3725+tags:
3726 - openstack
3727 provides:
3728 controller-api:
3729
3730=== added directory 'tests'
3731=== added file 'tests/015-basic-trusty-icehouse'
3732--- tests/015-basic-trusty-icehouse 1970-01-01 00:00:00 +0000
3733+++ tests/015-basic-trusty-icehouse 2015-11-11 19:55:10 +0000
3734@@ -0,0 +1,9 @@
3735+#!/usr/bin/python
3736+
3737+"""Amulet tests on a basic odl controller deployment on trusty-icehouse."""
3738+
3739+from basic_deployment import ODLControllerBasicDeployment
3740+
3741+if __name__ == '__main__':
3742+ deployment = ODLControllerBasicDeployment(series='trusty')
3743+ deployment.run_tests()
3744
3745=== added file 'tests/016-basic-trusty-juno'
3746--- tests/016-basic-trusty-juno 1970-01-01 00:00:00 +0000
3747+++ tests/016-basic-trusty-juno 2015-11-11 19:55:10 +0000
3748@@ -0,0 +1,11 @@
3749+#!/usr/bin/python
3750+
3751+"""Amulet tests on a basic odl controller deployment on trusty-juno."""
3752+
3753+from basic_deployment import ODLControllerBasicDeployment
3754+
3755+if __name__ == '__main__':
3756+ deployment = ODLControllerBasicDeployment(series='trusty',
3757+ openstack='cloud:trusty-juno',
3758+ source='cloud:trusty-updates/juno')
3759+ deployment.run_tests()
3760
3761=== added file 'tests/017-basic-trusty-kilo'
3762--- tests/017-basic-trusty-kilo 1970-01-01 00:00:00 +0000
3763+++ tests/017-basic-trusty-kilo 2015-11-11 19:55:10 +0000
3764@@ -0,0 +1,11 @@
3765+#!/usr/bin/python
3766+
3767+"""Amulet tests on a basic odl controller deployment on trusty-kilo."""
3768+
3769+from basic_deployment import ODLControllerBasicDeployment
3770+
3771+if __name__ == '__main__':
3772+ deployment = ODLControllerBasicDeployment(series='trusty',
3773+ openstack='cloud:trusty-kilo',
3774+ source='cloud:trusty-updates/kilo')
3775+ deployment.run_tests()
3776
3777=== added file 'tests/018-basic-trusty-liberty'
3778--- tests/018-basic-trusty-liberty 1970-01-01 00:00:00 +0000
3779+++ tests/018-basic-trusty-liberty 2015-11-11 19:55:10 +0000
3780@@ -0,0 +1,11 @@
3781+#!/usr/bin/python
3782+
3783+"""Amulet tests on a basic odl controller deployment on trusty-liberty."""
3784+
3785+from basic_deployment import ODLControllerBasicDeployment
3786+
3787+if __name__ == '__main__':
3788+ deployment = ODLControllerBasicDeployment(series='trusty',
3789+ openstack='cloud:trusty-liberty',
3790+ source='cloud:trusty-updates/liberty')
3791+ deployment.run_tests()
3792
3793=== added file 'tests/basic_deployment.py'
3794--- tests/basic_deployment.py 1970-01-01 00:00:00 +0000
3795+++ tests/basic_deployment.py 2015-11-11 19:55:10 +0000
3796@@ -0,0 +1,465 @@
3797+#!/usr/bin/python
3798+
3799+import amulet
3800+import os
3801+
3802+from neutronclient.v2_0 import client as neutronclient
3803+
3804+from charmhelpers.contrib.openstack.amulet.deployment import (
3805+ OpenStackAmuletDeployment
3806+)
3807+
3808+from charmhelpers.contrib.openstack.amulet.utils import (
3809+ OpenStackAmuletUtils,
3810+ DEBUG,
3811+ # ERROR
3812+)
3813+
3814+# Use DEBUG to turn on debug logging
3815+u = OpenStackAmuletUtils(DEBUG)
3816+
3817+
3818+class ODLControllerBasicDeployment(OpenStackAmuletDeployment):
3819+ """Amulet tests on a basic OVS ODL deployment."""
3820+
3821+ def __init__(self, series, openstack=None, source=None, git=False,
3822+ stable=False):
3823+ """Deploy the entire test environment."""
3824+ super(ODLControllerBasicDeployment, self).__init__(series, openstack,
3825+ source, stable)
3826+ self._add_services()
3827+ self._add_relations()
3828+ self._configure_services()
3829+ self._deploy()
3830+ exclude_services = ['mysql', 'odl-controller', 'neutron-api-odl']
3831+ self._auto_wait_for_status(exclude_services=exclude_services)
3832+ self._initialize_tests()
3833+
3834+ def _add_services(self):
3835+ """Add services
3836+
3837+ Add the services that we're testing, where odl-controller is local,
3838+ and the rest of the service are from lp branches that are
3839+ compatible with the local charm (e.g. stable or next).
3840+ """
3841+ this_service = {
3842+ 'name': 'odl-controller',
3843+ 'constraints': {'mem': '8G'},
3844+ }
3845+ other_services = [
3846+ {'name': 'mysql'},
3847+ {'name': 'rabbitmq-server'},
3848+ {'name': 'keystone'},
3849+ {'name': 'nova-cloud-controller'},
3850+ {'name': 'neutron-gateway'},
3851+ {
3852+ 'name': 'neutron-api-odl',
3853+ 'location': 'lp:~openstack-charmers/charms/trusty/'
3854+ 'neutron-api-odl/vpp',
3855+ },
3856+ {
3857+ 'name': 'openvswitch-odl',
3858+ 'location': 'lp:~openstack-charmers/charms/trusty/'
3859+ 'openvswitch-odl/trunk',
3860+ },
3861+ {'name': 'neutron-api'},
3862+ {'name': 'nova-compute'},
3863+ {'name': 'glance'},
3864+ ]
3865+
3866+ super(ODLControllerBasicDeployment, self)._add_services(
3867+ this_service, other_services)
3868+
3869+ def _add_relations(self):
3870+ """Add all of the relations for the services."""
3871+ relations = {
3872+ 'keystone:shared-db': 'mysql:shared-db',
3873+ 'neutron-gateway:shared-db': 'mysql:shared-db',
3874+ 'neutron-gateway:amqp': 'rabbitmq-server:amqp',
3875+ 'nova-cloud-controller:quantum-network-service':
3876+ 'neutron-gateway:quantum-network-service',
3877+ 'nova-cloud-controller:shared-db': 'mysql:shared-db',
3878+ 'nova-cloud-controller:identity-service': 'keystone:'
3879+ 'identity-service',
3880+ 'nova-cloud-controller:amqp': 'rabbitmq-server:amqp',
3881+ 'neutron-api:shared-db': 'mysql:shared-db',
3882+ 'neutron-api:amqp': 'rabbitmq-server:amqp',
3883+ 'neutron-api:neutron-api': 'nova-cloud-controller:neutron-api',
3884+ 'neutron-api:identity-service': 'keystone:identity-service',
3885+ 'neutron-api:neutron-plugin-api-subordinate':
3886+ 'neutron-api-odl:neutron-plugin-api-subordinate',
3887+ 'neutron-gateway:juju-info': 'openvswitch-odl:container',
3888+ 'openvswitch-odl:ovsdb-manager': 'odl-controller:ovsdb-manager',
3889+ 'neutron-api-odl:odl-controller': 'odl-controller:controller-api',
3890+ 'glance:identity-service': 'keystone:identity-service',
3891+ 'glance:shared-db': 'mysql:shared-db',
3892+ 'glance:amqp': 'rabbitmq-server:amqp',
3893+ 'nova-compute:image-service': 'glance:image-service',
3894+ 'nova-compute:shared-db': 'mysql:shared-db',
3895+ 'nova-compute:amqp': 'rabbitmq-server:amqp',
3896+ 'nova-cloud-controller:cloud-compute': 'nova-compute:'
3897+ 'cloud-compute',
3898+ 'nova-cloud-controller:image-service': 'glance:image-service',
3899+ }
3900+ super(ODLControllerBasicDeployment, self)._add_relations(relations)
3901+
3902+ def _configure_services(self):
3903+ """Configure all of the services."""
3904+ neutron_gateway_config = {'plugin': 'ovs-odl',
3905+ 'instance-mtu': '1400'}
3906+ neutron_api_config = {'neutron-security-groups': 'False',
3907+ 'manage-neutron-plugin-legacy-mode': 'False'}
3908+ neutron_api_odl_config = {'overlay-network-type': 'vxlan gre'}
3909+ odl_controller_config = {}
3910+ if os.environ.get('AMULET_ODL_LOCATION'):
3911+ odl_controller_config['install-url'] = \
3912+ os.environ['AMULET_ODL_LOCATION']
3913+ if os.environ.get('AMULET_HTTP_PROXY'):
3914+ odl_controller_config['http-proxy'] = \
3915+ os.environ['AMULET_HTTP_PROXY']
3916+ if os.environ.get('AMULET_HTTP_PROXY'):
3917+ odl_controller_config['https-proxy'] = \
3918+ os.environ['AMULET_HTTP_PROXY']
3919+ keystone_config = {'admin-password': 'openstack',
3920+ 'admin-token': 'ubuntutesting'}
3921+ nova_cc_config = {'network-manager': 'Quantum',
3922+ 'quantum-security-groups': 'yes'}
3923+ configs = {'neutron-gateway': neutron_gateway_config,
3924+ 'neutron-api': neutron_api_config,
3925+ 'neutron-api-odl': neutron_api_odl_config,
3926+ 'odl-controller': odl_controller_config,
3927+ 'keystone': keystone_config,
3928+ 'nova-cloud-controller': nova_cc_config}
3929+ super(ODLControllerBasicDeployment, self)._configure_services(configs)
3930+
3931+ def _initialize_tests(self):
3932+ """Perform final initialization before tests get run."""
3933+ # Access the sentries for inspecting service units
3934+ self.mysql_sentry = self.d.sentry['mysql'][0]
3935+ self.keystone_sentry = self.d.sentry['keystone'][0]
3936+ self.rmq_sentry = self.d.sentry['rabbitmq-server'][0]
3937+ self.nova_cc_sentry = self.d.sentry['nova-cloud-controller'][0]
3938+ self.neutron_gateway_sentry = self.d.sentry['neutron-gateway'][0]
3939+ self.neutron_api_sentry = self.d.sentry['neutron-api'][0]
3940+ self.odl_controller_sentry = self.d.sentry['odl-controller'][0]
3941+ self.neutron_api_odl_sentry = self.d.sentry['neutron-api-odl'][0]
3942+ self.openvswitch_odl_sentry = self.d.sentry['openvswitch-odl'][0]
3943+
3944+ # Authenticate admin with keystone
3945+ self.keystone = u.authenticate_keystone_admin(self.keystone_sentry,
3946+ user='admin',
3947+ password='openstack',
3948+ tenant='admin')
3949+
3950+ # Authenticate admin with neutron
3951+ ep = self.keystone.service_catalog.url_for(service_type='identity',
3952+ endpoint_type='publicURL')
3953+ self.neutron = neutronclient.Client(auth_url=ep,
3954+ username='admin',
3955+ password='openstack',
3956+ tenant_name='admin',
3957+ region_name='RegionOne')
3958+ # Authenticate admin with glance endpoint
3959+ self.glance = u.authenticate_glance_admin(self.keystone)
3960+ # Create a demo tenant/role/user
3961+ self.demo_tenant = 'demoTenant'
3962+ self.demo_role = 'demoRole'
3963+ self.demo_user = 'demoUser'
3964+ if not u.tenant_exists(self.keystone, self.demo_tenant):
3965+ tenant = self.keystone.tenants.create(tenant_name=self.demo_tenant,
3966+ description='demo tenant',
3967+ enabled=True)
3968+ self.keystone.roles.create(name=self.demo_role)
3969+ self.keystone.users.create(name=self.demo_user,
3970+ password='password',
3971+ tenant_id=tenant.id,
3972+ email='demo@demo.com')
3973+
3974+ # Authenticate demo user with keystone
3975+ self.keystone_demo = \
3976+ u.authenticate_keystone_user(self.keystone, user=self.demo_user,
3977+ password='password',
3978+ tenant=self.demo_tenant)
3979+
3980+ # Authenticate demo user with nova-api
3981+ self.nova_demo = u.authenticate_nova_user(self.keystone,
3982+ user=self.demo_user,
3983+ password='password',
3984+ tenant=self.demo_tenant)
3985+
3986+ def test_100_services(self):
3987+ """Verify the expected services are running on the corresponding
3988+ service units."""
3989+ neutron_services = ['neutron-dhcp-agent',
3990+ 'neutron-lbaas-agent',
3991+ 'neutron-metadata-agent',
3992+ 'neutron-metering-agent',
3993+ 'neutron-l3-agent']
3994+
3995+ nova_cc_services = ['nova-api-ec2',
3996+ 'nova-api-os-compute',
3997+ 'nova-objectstore',
3998+ 'nova-cert',
3999+ 'nova-scheduler',
4000+ 'nova-conductor']
4001+
4002+ odl_c_services = ['odl-controller']
4003+
4004+ commands = {
4005+ self.mysql_sentry: ['mysql'],
4006+ self.keystone_sentry: ['keystone'],
4007+ self.nova_cc_sentry: nova_cc_services,
4008+ self.neutron_gateway_sentry: neutron_services,
4009+ self.odl_controller_sentry: odl_c_services,
4010+ }
4011+
4012+ ret = u.validate_services_by_name(commands)
4013+ if ret:
4014+ amulet.raise_status(amulet.FAIL, msg=ret)
4015+
4016+ def test_102_service_catalog(self):
4017+ """Verify that the service catalog endpoint data is valid."""
4018+ u.log.debug('Checking keystone service catalog...')
4019+ endpoint_check = {
4020+ 'adminURL': u.valid_url,
4021+ 'id': u.not_null,
4022+ 'region': 'RegionOne',
4023+ 'publicURL': u.valid_url,
4024+ 'internalURL': u.valid_url
4025+ }
4026+ expected = {
4027+ 'network': [endpoint_check],
4028+ 'compute': [endpoint_check],
4029+ 'identity': [endpoint_check]
4030+ }
4031+ actual = self.keystone.service_catalog.get_endpoints()
4032+
4033+ ret = u.validate_svc_catalog_endpoint_data(expected, actual)
4034+ if ret:
4035+ amulet.raise_status(amulet.FAIL, msg=ret)
4036+
4037+ def test_104_network_endpoint(self):
4038+ """Verify the neutron network endpoint data."""
4039+ u.log.debug('Checking neutron network api endpoint data...')
4040+ endpoints = self.keystone.endpoints.list()
4041+ admin_port = internal_port = public_port = '9696'
4042+ expected = {
4043+ 'id': u.not_null,
4044+ 'region': 'RegionOne',
4045+ 'adminurl': u.valid_url,
4046+ 'internalurl': u.valid_url,
4047+ 'publicurl': u.valid_url,
4048+ 'service_id': u.not_null
4049+ }
4050+ ret = u.validate_endpoint_data(endpoints, admin_port, internal_port,
4051+ public_port, expected)
4052+
4053+ if ret:
4054+ amulet.raise_status(amulet.FAIL,
4055+ msg='glance endpoint: {}'.format(ret))
4056+
4057+ def test_110_users(self):
4058+ """Verify expected users."""
4059+ u.log.debug('Checking keystone users...')
4060+ expected = [
4061+ {'name': 'admin',
4062+ 'enabled': True,
4063+ 'tenantId': u.not_null,
4064+ 'id': u.not_null,
4065+ 'email': 'juju@localhost'},
4066+ {'name': 'quantum',
4067+ 'enabled': True,
4068+ 'tenantId': u.not_null,
4069+ 'id': u.not_null,
4070+ 'email': 'juju@localhost'}
4071+ ]
4072+
4073+ if self._get_openstack_release() >= self.trusty_kilo:
4074+ # Kilo or later
4075+ expected.append({
4076+ 'name': 'nova',
4077+ 'enabled': True,
4078+ 'tenantId': u.not_null,
4079+ 'id': u.not_null,
4080+ 'email': 'juju@localhost'
4081+ })
4082+ else:
4083+ # Juno and earlier
4084+ expected.append({
4085+ 'name': 's3_ec2_nova',
4086+ 'enabled': True,
4087+ 'tenantId': u.not_null,
4088+ 'id': u.not_null,
4089+ 'email': 'juju@localhost'
4090+ })
4091+
4092+ actual = self.keystone.users.list()
4093+ ret = u.validate_user_data(expected, actual)
4094+ if ret:
4095+ amulet.raise_status(amulet.FAIL, msg=ret)
4096+
4097+ def test_200_odl_controller_controller_api_relation(self):
4098+ """Verify the odl-controller to neutron-api-odl relation data"""
4099+ u.log.debug('Checking odl-controller to neutron-api-odl relation data')
4100+ unit = self.odl_controller_sentry
4101+ relation = ['controller-api', 'neutron-api-odl:odl-controller']
4102+ expected = {
4103+ 'private-address': u.valid_ip,
4104+ 'username': 'admin',
4105+ 'password': 'admin',
4106+ 'port': '8080',
4107+ }
4108+
4109+ ret = u.validate_relation_data(unit, relation, expected)
4110+ if ret:
4111+ message = u.relation_error('odl-controller controller-api', ret)
4112+ amulet.raise_status(amulet.FAIL, msg=message)
4113+
4114+ def test_201_neutron_api_odl_odl_controller_relation(self):
4115+ """Verify the odl-controller to neutron-api-odl relation data"""
4116+ u.log.debug('Checking odl-controller to neutron-api-odl relation data')
4117+ unit = self.neutron_api_odl_sentry
4118+ relation = ['odl-controller', 'odl-controller:controller-api']
4119+ expected = {
4120+ 'private-address': u.valid_ip,
4121+ }
4122+
4123+ ret = u.validate_relation_data(unit, relation, expected)
4124+ if ret:
4125+ message = u.relation_error('neutron-api-odl odl-controller', ret)
4126+ amulet.raise_status(amulet.FAIL, msg=message)
4127+
4128+ def test_202_odl_controller_ovsdb_manager_relation(self):
4129+ """Verify the odl-controller to openvswitch-odl relation data"""
4130+ u.log.debug('Checking odl-controller to openvswitch-odl relation data')
4131+ unit = self.odl_controller_sentry
4132+ relation = ['ovsdb-manager', 'openvswitch-odl:ovsdb-manager']
4133+ expected = {
4134+ 'private-address': u.valid_ip,
4135+ 'protocol': 'tcp',
4136+ 'port': '6640',
4137+ }
4138+
4139+ ret = u.validate_relation_data(unit, relation, expected)
4140+ if ret:
4141+ message = u.relation_error('odl-controller openvswitch-odl', ret)
4142+ amulet.raise_status(amulet.FAIL, msg=message)
4143+
4144+ def test_203_openvswitch_odl_ovsdb_manager_relation(self):
4145+ """Verify the openvswitch-odl to odl-controller relation data"""
4146+ u.log.debug('Checking openvswitch-odl to odl-controller relation data')
4147+ unit = self.openvswitch_odl_sentry
4148+ relation = ['ovsdb-manager', 'odl-controller:ovsdb-manager']
4149+ expected = {
4150+ 'private-address': u.valid_ip,
4151+ }
4152+
4153+ ret = u.validate_relation_data(unit, relation, expected)
4154+ if ret:
4155+ message = u.relation_error('openvswitch-odl to odl-controller',
4156+ ret)
4157+ amulet.raise_status(amulet.FAIL, msg=message)
4158+
4159+ def test_400_create_network(self):
4160+ """Create a network, verify that it exists, and then delete it."""
4161+ u.log.debug('Creating neutron network...')
4162+ self.neutron.format = 'json'
4163+ net_name = 'ext_net'
4164+
4165+ # Verify that the network doesn't exist
4166+ networks = self.neutron.list_networks(name=net_name)
4167+ net_count = len(networks['networks'])
4168+ if net_count != 0:
4169+ msg = "Expected zero networks, found {}".format(net_count)
4170+ amulet.raise_status(amulet.FAIL, msg=msg)
4171+
4172+ # Create a network and verify that it exists
4173+ network = {'name': net_name}
4174+ self.neutron.create_network({'network': network})
4175+
4176+ networks = self.neutron.list_networks(name=net_name)
4177+ u.log.debug('Networks: {}'.format(networks))
4178+ net_len = len(networks['networks'])
4179+ if net_len != 1:
4180+ msg = "Expected 1 network, found {}".format(net_len)
4181+ amulet.raise_status(amulet.FAIL, msg=msg)
4182+
4183+ u.log.debug('Confirming new neutron network...')
4184+ network = networks['networks'][0]
4185+ if network['name'] != net_name:
4186+ amulet.raise_status(amulet.FAIL, msg="network ext_net not found")
4187+
4188+ # Cleanup
4189+ u.log.debug('Deleting neutron network...')
4190+ self.neutron.delete_network(network['id'])
4191+
4192+ def test_400_gateway_bridges(self):
4193+ """Ensure that all bridges are present and configured with the
4194+ ODL controller as their NorthBound controller URL."""
4195+ odl_ip = self.odl_controller_sentry.relation(
4196+ 'ovsdb-manager',
4197+ 'openvswitch-odl:ovsdb-manager'
4198+ )['private-address']
4199+ controller_url = "tcp:{}:6633".format(odl_ip)
4200+ cmd = 'ovs-vsctl list-br'
4201+ output, _ = self.neutron_gateway_sentry.run(cmd)
4202+ bridges = output.split()
4203+ u.log.debug('Checking bridge configuration...')
4204+ for bridge in ['br-int', 'br-ex', 'br-data']:
4205+ if bridge not in bridges:
4206+ amulet.raise_status(
4207+ amulet.FAIL,
4208+ msg="Missing bridge {} from gateway unit".format(bridge)
4209+ )
4210+ cmd = 'ovs-vsctl get-controller {}'.format(bridge)
4211+ br_controllers, _ = self.neutron_gateway_sentry.run(cmd)
4212+ br_controllers = list(set(br_controllers.split('\n')))
4213+ if len(br_controllers) != 1 or br_controllers[0] != controller_url:
4214+ status, _ = self.neutron_gateway_sentry.run('ovs-vsctl show')
4215+ amulet.raise_status(
4216+ amulet.FAIL,
4217+ msg="Controller configuration on bridge"
4218+ " {} incorrect: !{}! != !{}!\n"
4219+ "{}".format(bridge,
4220+ br_controllers,
4221+ controller_url,
4222+ status)
4223+ )
4224+
4225+ def test_400_image_instance_create(self):
4226+ """Create an image/instance, verify they exist, and delete them."""
4227+ # NOTE(coreycb): Skipping failing test on essex until resolved. essex
4228+ # nova API calls are getting "Malformed request url
4229+ # (HTTP 400)".
4230+ if self._get_openstack_release() == self.precise_essex:
4231+ u.log.error("Skipping test (due to Essex)")
4232+ return
4233+
4234+ u.log.debug('Checking nova instance creation...')
4235+
4236+ image = u.create_cirros_image(self.glance, "cirros-image")
4237+ if not image:
4238+ amulet.raise_status(amulet.FAIL, msg="Image create failed")
4239+
4240+ instance = u.create_instance(self.nova_demo, "cirros-image", "cirros",
4241+ "m1.tiny")
4242+ if not instance:
4243+ amulet.raise_status(amulet.FAIL, msg="Instance create failed")
4244+
4245+ found = False
4246+ for instance in self.nova_demo.servers.list():
4247+ if instance.name == 'cirros':
4248+ found = True
4249+ if instance.status != 'ACTIVE':
4250+ msg = "cirros instance is not active"
4251+ amulet.raise_status(amulet.FAIL, msg=msg)
4252+
4253+ if not found:
4254+ message = "nova cirros instance does not exist"
4255+ amulet.raise_status(amulet.FAIL, msg=message)
4256+
4257+ u.delete_resource(self.glance.images, image.id,
4258+ msg="glance image")
4259+
4260+ u.delete_resource(self.nova_demo.servers, instance.id,
4261+ msg="nova instance")
4262
4263=== added directory 'tests/charmhelpers'
4264=== added file 'tests/charmhelpers/__init__.py'
4265--- tests/charmhelpers/__init__.py 1970-01-01 00:00:00 +0000
4266+++ tests/charmhelpers/__init__.py 2015-11-11 19:55:10 +0000
4267@@ -0,0 +1,38 @@
4268+# Copyright 2014-2015 Canonical Limited.
4269+#
4270+# This file is part of charm-helpers.
4271+#
4272+# charm-helpers is free software: you can redistribute it and/or modify
4273+# it under the terms of the GNU Lesser General Public License version 3 as
4274+# published by the Free Software Foundation.
4275+#
4276+# charm-helpers is distributed in the hope that it will be useful,
4277+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4278+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4279+# GNU Lesser General Public License for more details.
4280+#
4281+# You should have received a copy of the GNU Lesser General Public License
4282+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4283+
4284+# Bootstrap charm-helpers, installing its dependencies if necessary using
4285+# only standard libraries.
4286+import subprocess
4287+import sys
4288+
4289+try:
4290+ import six # flake8: noqa
4291+except ImportError:
4292+ if sys.version_info.major == 2:
4293+ subprocess.check_call(['apt-get', 'install', '-y', 'python-six'])
4294+ else:
4295+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-six'])
4296+ import six # flake8: noqa
4297+
4298+try:
4299+ import yaml # flake8: noqa
4300+except ImportError:
4301+ if sys.version_info.major == 2:
4302+ subprocess.check_call(['apt-get', 'install', '-y', 'python-yaml'])
4303+ else:
4304+ subprocess.check_call(['apt-get', 'install', '-y', 'python3-yaml'])
4305+ import yaml # flake8: noqa
4306
4307=== added directory 'tests/charmhelpers/contrib'
4308=== added file 'tests/charmhelpers/contrib/__init__.py'
4309--- tests/charmhelpers/contrib/__init__.py 1970-01-01 00:00:00 +0000
4310+++ tests/charmhelpers/contrib/__init__.py 2015-11-11 19:55:10 +0000
4311@@ -0,0 +1,15 @@
4312+# Copyright 2014-2015 Canonical Limited.
4313+#
4314+# This file is part of charm-helpers.
4315+#
4316+# charm-helpers is free software: you can redistribute it and/or modify
4317+# it under the terms of the GNU Lesser General Public License version 3 as
4318+# published by the Free Software Foundation.
4319+#
4320+# charm-helpers is distributed in the hope that it will be useful,
4321+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4322+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4323+# GNU Lesser General Public License for more details.
4324+#
4325+# You should have received a copy of the GNU Lesser General Public License
4326+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4327
4328=== added directory 'tests/charmhelpers/contrib/amulet'
4329=== added file 'tests/charmhelpers/contrib/amulet/__init__.py'
4330--- tests/charmhelpers/contrib/amulet/__init__.py 1970-01-01 00:00:00 +0000
4331+++ tests/charmhelpers/contrib/amulet/__init__.py 2015-11-11 19:55:10 +0000
4332@@ -0,0 +1,15 @@
4333+# Copyright 2014-2015 Canonical Limited.
4334+#
4335+# This file is part of charm-helpers.
4336+#
4337+# charm-helpers is free software: you can redistribute it and/or modify
4338+# it under the terms of the GNU Lesser General Public License version 3 as
4339+# published by the Free Software Foundation.
4340+#
4341+# charm-helpers is distributed in the hope that it will be useful,
4342+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4343+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4344+# GNU Lesser General Public License for more details.
4345+#
4346+# You should have received a copy of the GNU Lesser General Public License
4347+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4348
4349=== added file 'tests/charmhelpers/contrib/amulet/deployment.py'
4350--- tests/charmhelpers/contrib/amulet/deployment.py 1970-01-01 00:00:00 +0000
4351+++ tests/charmhelpers/contrib/amulet/deployment.py 2015-11-11 19:55:10 +0000
4352@@ -0,0 +1,95 @@
4353+# Copyright 2014-2015 Canonical Limited.
4354+#
4355+# This file is part of charm-helpers.
4356+#
4357+# charm-helpers is free software: you can redistribute it and/or modify
4358+# it under the terms of the GNU Lesser General Public License version 3 as
4359+# published by the Free Software Foundation.
4360+#
4361+# charm-helpers is distributed in the hope that it will be useful,
4362+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4363+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4364+# GNU Lesser General Public License for more details.
4365+#
4366+# You should have received a copy of the GNU Lesser General Public License
4367+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4368+
4369+import amulet
4370+import os
4371+import six
4372+
4373+
4374+class AmuletDeployment(object):
4375+ """Amulet deployment.
4376+
4377+ This class provides generic Amulet deployment and test runner
4378+ methods.
4379+ """
4380+
4381+ def __init__(self, series=None):
4382+ """Initialize the deployment environment."""
4383+ self.series = None
4384+
4385+ if series:
4386+ self.series = series
4387+ self.d = amulet.Deployment(series=self.series)
4388+ else:
4389+ self.d = amulet.Deployment()
4390+
4391+ def _add_services(self, this_service, other_services):
4392+ """Add services.
4393+
4394+ Add services to the deployment where this_service is the local charm
4395+ that we're testing and other_services are the other services that
4396+ are being used in the local amulet tests.
4397+ """
4398+ if this_service['name'] != os.path.basename(os.getcwd()):
4399+ s = this_service['name']
4400+ msg = "The charm's root directory name needs to be {}".format(s)
4401+ amulet.raise_status(amulet.FAIL, msg=msg)
4402+
4403+ if 'units' not in this_service:
4404+ this_service['units'] = 1
4405+
4406+ self.d.add(this_service['name'], units=this_service['units'],
4407+ constraints=this_service.get('constraints'))
4408+
4409+ for svc in other_services:
4410+ if 'location' in svc:
4411+ branch_location = svc['location']
4412+ elif self.series:
4413+ branch_location = 'cs:{}/{}'.format(self.series, svc['name']),
4414+ else:
4415+ branch_location = None
4416+
4417+ if 'units' not in svc:
4418+ svc['units'] = 1
4419+
4420+ self.d.add(svc['name'], charm=branch_location, units=svc['units'],
4421+ constraints=svc.get('constraints'))
4422+
4423+ def _add_relations(self, relations):
4424+ """Add all of the relations for the services."""
4425+ for k, v in six.iteritems(relations):
4426+ self.d.relate(k, v)
4427+
4428+ def _configure_services(self, configs):
4429+ """Configure all of the services."""
4430+ for service, config in six.iteritems(configs):
4431+ self.d.configure(service, config)
4432+
4433+ def _deploy(self):
4434+ """Deploy environment and wait for all hooks to finish executing."""
4435+ try:
4436+ self.d.setup(timeout=900)
4437+ self.d.sentry.wait(timeout=900)
4438+ except amulet.helpers.TimeoutError:
4439+ amulet.raise_status(amulet.FAIL, msg="Deployment timed out")
4440+ except Exception:
4441+ raise
4442+
4443+ def run_tests(self):
4444+ """Run all of the methods that are prefixed with 'test_'."""
4445+ for test in dir(self):
4446+ if test.startswith('test_'):
4447+ getattr(self, test)()
4448
4449=== added file 'tests/charmhelpers/contrib/amulet/utils.py'
4450--- tests/charmhelpers/contrib/amulet/utils.py 1970-01-01 00:00:00 +0000
4451+++ tests/charmhelpers/contrib/amulet/utils.py 2015-11-11 19:55:10 +0000
4452@@ -0,0 +1,818 @@
4453+# Copyright 2014-2015 Canonical Limited.
4454+#
4455+# This file is part of charm-helpers.
4456+#
4457+# charm-helpers is free software: you can redistribute it and/or modify
4458+# it under the terms of the GNU Lesser General Public License version 3 as
4459+# published by the Free Software Foundation.
4460+#
4461+# charm-helpers is distributed in the hope that it will be useful,
4462+# but WITHOUT ANY WARRANTY; without even the implied warranty of
4463+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4464+# GNU Lesser General Public License for more details.
4465+#
4466+# You should have received a copy of the GNU Lesser General Public License
4467+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
4468+
4469+import io
4470+import json
4471+import logging
4472+import os
4473+import re
4474+import socket
4475+import subprocess
4476+import sys
4477+import time
4478+import uuid
4479+
4480+import amulet
4481+import distro_info
4482+import six
4483+from six.moves import configparser
4484+if six.PY3:
4485+ from urllib import parse as urlparse
4486+else:
4487+ import urlparse
4488+
4489+
4490+class AmuletUtils(object):
4491+ """Amulet utilities.
4492+
4493+ This class provides common utility functions that are used by Amulet
4494+ tests.
4495+ """
4496+
4497+ def __init__(self, log_level=logging.ERROR):
4498+ self.log = self.get_logger(level=log_level)
4499+ self.ubuntu_releases = self.get_ubuntu_releases()
4500+
4501+ def get_logger(self, name="amulet-logger", level=logging.DEBUG):
4502+ """Get a logger object that will log to stdout."""
4503+ log = logging
4504+ logger = log.getLogger(name)
4505+ fmt = log.Formatter("%(asctime)s %(funcName)s "
4506+ "%(levelname)s: %(message)s")
4507+
4508+ handler = log.StreamHandler(stream=sys.stdout)
4509+ handler.setLevel(level)
4510+ handler.setFormatter(fmt)
4511+
4512+ logger.addHandler(handler)
4513+ logger.setLevel(level)
4514+
4515+ return logger
4516+
4517+ def valid_ip(self, ip):
4518+ if re.match(r"^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$", ip):
4519+ return True
4520+ else:
4521+ return False
4522+
4523+ def valid_url(self, url):
4524+ p = re.compile(
4525+ r'^(?:http|ftp)s?://'
4526+ r'(?:(?:[A-Z0-9](?:[A-Z0-9-]{0,61}[A-Z0-9])?\.)+(?:[A-Z]{2,6}\.?|[A-Z0-9-]{2,}\.?)|' # noqa
4527+ r'localhost|'
4528+ r'\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})'
4529+ r'(?::\d+)?'
4530+ r'(?:/?|[/?]\S+)$',
4531+ re.IGNORECASE)
4532+ if p.match(url):
4533+ return True
4534+ else:
4535+ return False
4536+
4537+ def get_ubuntu_release_from_sentry(self, sentry_unit):
4538+ """Get Ubuntu release codename from sentry unit.
4539+
4540+ :param sentry_unit: amulet sentry/service unit pointer
4541+ :returns: list of strings - release codename, failure message
4542+ """
4543+ msg = None
4544+ cmd = 'lsb_release -cs'
4545+ release, code = sentry_unit.run(cmd)
4546+ if code == 0:
4547+ self.log.debug('{} lsb_release: {}'.format(
4548+ sentry_unit.info['unit_name'], release))
4549+ else:
4550+ msg = ('{} `{}` returned {} '
4551+ '{}'.format(sentry_unit.info['unit_name'],
4552+ cmd, release, code))
4553+ if release not in self.ubuntu_releases:
4554+ msg = ("Release ({}) not found in Ubuntu releases "
4555+ "({})".format(release, self.ubuntu_releases))
4556+ return release, msg
4557+
4558+ def validate_services(self, commands):
4559+ """Validate that lists of commands succeed on service units. Can be
4560+ used to verify system services are running on the corresponding
4561+ service units.
4562+
4563+ :param commands: dict with sentry keys and arbitrary command list vals
4564+ :returns: None if successful, Failure string message otherwise
4565+ """
4566+ self.log.debug('Checking status of system services...')
4567+
4568+ # /!\ DEPRECATION WARNING (beisner):
4569+ # New and existing tests should be rewritten to use
4570+ # validate_services_by_name() as it is aware of init systems.
4571+ self.log.warn('DEPRECATION WARNING: use '
4572+ 'validate_services_by_name instead of validate_services '
4573+ 'due to init system differences.')
4574+
4575+ for k, v in six.iteritems(commands):
4576+ for cmd in v:
4577+ output, code = k.run(cmd)
4578+ self.log.debug('{} `{}` returned '
4579+ '{}'.format(k.info['unit_name'],
4580+ cmd, code))
4581+ if code != 0:
4582+ return "command `{}` returned {}".format(cmd, str(code))
4583+ return None
4584+
4585+ def validate_services_by_name(self, sentry_services):
4586+ """Validate system service status by service name, automatically
4587+ detecting init system based on Ubuntu release codename.
4588+
4589+ :param sentry_services: dict with sentry keys and svc list values
4590+ :returns: None if successful, Failure string message otherwise
4591+ """
4592+ self.log.debug('Checking status of system services...')
4593+
4594+ # Point at which systemd became a thing
4595+ systemd_switch = self.ubuntu_releases.index('vivid')
4596+
4597+ for sentry_unit, services_list in six.iteritems(sentry_services):
4598+ # Get lsb_release codename from unit
4599+ release, ret = self.get_ubuntu_release_from_sentry(sentry_unit)
4600+ if ret:
4601+ return ret
4602+
4603+ for service_name in services_list:
4604+ if (self.ubuntu_releases.index(release) >= systemd_switch or
4605+ service_name in ['rabbitmq-server', 'apache2']):
4606+ # init is systemd (or regular sysv)
4607+ cmd = 'sudo service {} status'.format(service_name)
4608+ output, code = sentry_unit.run(cmd)
4609+ service_running = code == 0
4610+ elif self.ubuntu_releases.index(release) < systemd_switch:
4611+ # init is upstart
4612+ cmd = 'sudo status {}'.format(service_name)
4613+ output, code = sentry_unit.run(cmd)
4614+ service_running = code == 0 and "start/running" in output
4615+
4616+ self.log.debug('{} `{}` returned '
4617+ '{}'.format(sentry_unit.info['unit_name'],
4618+ cmd, code))
4619+ if not service_running:
4620+ return u"command `{}` returned {} {}".format(
4621+ cmd, output, str(code))
4622+ return None
4623+
4624+ def _get_config(self, unit, filename):
4625+ """Get a ConfigParser object for parsing a unit's config file."""
4626+ file_contents = unit.file_contents(filename)
4627+
4628+ # NOTE(beisner): by default, ConfigParser does not handle options
4629+ # with no value, such as the flags used in the mysql my.cnf file.
4630+ # https://bugs.python.org/issue7005
4631+ config = configparser.ConfigParser(allow_no_value=True)
4632+ config.readfp(io.StringIO(file_contents))
4633+ return config
4634+
4635+ def validate_config_data(self, sentry_unit, config_file, section,
4636+ expected):
4637+ """Validate config file data.
4638+
4639+ Verify that the specified section of the config file contains
4640+ the expected option key:value pairs.
4641+
4642+ Compare expected dictionary data vs actual dictionary data.
4643+ The values in the 'expected' dictionary can be strings, bools, ints,
4644+ longs, or can be a function that evaluates a variable and returns a
4645+ bool.
4646+ """
4647+ self.log.debug('Validating config file data ({} in {} on {})'
4648+ '...'.format(section, config_file,
4649+ sentry_unit.info['unit_name']))
4650+ config = self._get_config(sentry_unit, config_file)
4651+
4652+ if section != 'DEFAULT' and not config.has_section(section):
4653+ return "section [{}] does not exist".format(section)
4654+
4655+ for k in expected.keys():
4656+ if not config.has_option(section, k):
4657+ return "section [{}] is missing option {}".format(section, k)
4658+
4659+ actual = config.get(section, k)
4660+ v = expected[k]
4661+ if (isinstance(v, six.string_types) or
4662+ isinstance(v, bool) or
4663+ isinstance(v, six.integer_types)):
4664+ # handle explicit values
4665+ if actual != v:
4666+ return "section [{}] {}:{} != expected {}:{}".format(
4667+ section, k, actual, k, expected[k])
4668+ # handle function pointers, such as not_null or valid_ip
4669+ elif not v(actual):
4670+ return "section [{}] {}:{} != expected {}:{}".format(
4671+ section, k, actual, k, expected[k])
4672+ return None
4673+
4674+ def _validate_dict_data(self, expected, actual):
4675+ """Validate dictionary data.
4676+
4677+ Compare expected dictionary data vs actual dictionary data.
4678+ The values in the 'expected' dictionary can be strings, bools, ints,
4679+ longs, or can be a function that evaluates a variable and returns a
4680+ bool.
4681+ """
4682+ self.log.debug('actual: {}'.format(repr(actual)))
4683+ self.log.debug('expected: {}'.format(repr(expected)))
4684+
4685+ for k, v in six.iteritems(expected):
4686+ if k in actual:
4687+ if (isinstance(v, six.string_types) or
4688+ isinstance(v, bool) or
4689+ isinstance(v, six.integer_types)):
4690+ # handle explicit values
4691+ if v != actual[k]:
4692+ return "{}:{}".format(k, actual[k])
4693+ # handle function pointers, such as not_null or valid_ip
4694+ elif not v(actual[k]):
4695+ return "{}:{}".format(k, actual[k])
4696+ else:
4697+ return "key '{}' does not exist".format(k)
4698+ return None
4699+
4700+ def validate_relation_data(self, sentry_unit, relation, expected):
4701+ """Validate actual relation data based on expected relation data."""
4702+ actual = sentry_unit.relation(relation[0], relation[1])
4703+ return self._validate_dict_data(expected, actual)
4704+
4705+ def _validate_list_data(self, expected, actual):
4706+ """Compare expected list vs actual list data."""
4707+ for e in expected:
4708+ if e not in actual:
4709+ return "expected item {} not found in actual list".format(e)
4710+ return None
4711+
4712+ def not_null(self, string):
4713+ if string is not None:
4714+ return True
4715+ else:
4716+ return False
4717+
4718+ def _get_file_mtime(self, sentry_unit, filename):
4719+ """Get last modification time of file."""
4720+ return sentry_unit.file_stat(filename)['mtime']
4721+
4722+ def _get_dir_mtime(self, sentry_unit, directory):
4723+ """Get last modification time of directory."""
4724+ return sentry_unit.directory_stat(directory)['mtime']
4725+
4726+ def _get_proc_start_time(self, sentry_unit, service, pgrep_full=None):
4727+ """Get start time of a process based on the last modification time
4728+ of the /proc/pid directory.
4729+
4730+ :sentry_unit: The sentry unit to check for the service on
4731+ :service: service name to look for in process table
4732+ :pgrep_full: [Deprecated] Use full command line search mode with pgrep
4733+ :returns: epoch time of service process start
4734+ :param commands: list of bash commands
4735+ :param sentry_units: list of sentry unit pointers
4736+ :returns: None if successful; Failure message otherwise
4737+ """
4738+ if pgrep_full is not None:
4739+ # /!\ DEPRECATION WARNING (beisner):
4740+ # No longer implemented, as pidof is now used instead of pgrep.
4741+ # https://bugs.launchpad.net/charm-helpers/+bug/1474030
4742+ self.log.warn('DEPRECATION WARNING: pgrep_full bool is no '
4743+ 'longer implemented re: lp 1474030.')
4744+
4745+ pid_list = self.get_process_id_list(sentry_unit, service)
4746+ pid = pid_list[0]
4747+ proc_dir = '/proc/{}'.format(pid)
4748+ self.log.debug('Pid for {} on {}: {}'.format(
4749+ service, sentry_unit.info['unit_name'], pid))
4750+
4751+ return self._get_dir_mtime(sentry_unit, proc_dir)
4752+
4753+ def service_restarted(self, sentry_unit, service, filename,
4754+ pgrep_full=None, sleep_time=20):
4755+ """Check if service was restarted.
4756+
4757+ Compare a service's start time vs a file's last modification time
4758+ (such as a config file for that service) to determine if the service
4759+ has been restarted.
4760+ """
4761+ # /!\ DEPRECATION WARNING (beisner):
4762+ # This method is prone to races in that no before-time is known.
4763+ # Use validate_service_config_changed instead.
4764+
4765+ # NOTE(beisner) pgrep_full is no longer implemented, as pidof is now
4766+ # used instead of pgrep. pgrep_full is still passed through to ensure
4767+ # deprecation WARNS. lp1474030
4768+ self.log.warn('DEPRECATION WARNING: use '
4769+ 'validate_service_config_changed instead of '
4770+ 'service_restarted due to known races.')
4771+
4772+ time.sleep(sleep_time)
4773+ if (self._get_proc_start_time(sentry_unit, service, pgrep_full) >=
4774+ self._get_file_mtime(sentry_unit, filename)):
4775+ return True
4776+ else:
4777+ return False
4778+
4779+ def service_restarted_since(self, sentry_unit, mtime, service,
4780+ pgrep_full=None, sleep_time=20,
4781+ retry_count=30, retry_sleep_time=10):
4782+ """Check if service was been started after a given time.
4783+
4784+ Args:
4785+ sentry_unit (sentry): The sentry unit to check for the service on
4786+ mtime (float): The epoch time to check against
4787+ service (string): service name to look for in process table
4788+ pgrep_full: [Deprecated] Use full command line search mode with pgrep
4789+ sleep_time (int): Initial sleep time (s) before looking for file
4790+ retry_sleep_time (int): Time (s) to sleep between retries
4791+ retry_count (int): If file is not found, how many times to retry
4792+
4793+ Returns:
4794+ bool: True if service found and its start time it newer than mtime,
4795+ False if service is older than mtime or if service was
4796+ not found.
4797+ """
4798+ # NOTE(beisner) pgrep_full is no longer implemented, as pidof is now
4799+ # used instead of pgrep. pgrep_full is still passed through to ensure
4800+ # deprecation WARNS. lp1474030
4801+
4802+ unit_name = sentry_unit.info['unit_name']
4803+ self.log.debug('Checking that %s service restarted since %s on '
4804+ '%s' % (service, mtime, unit_name))
4805+ time.sleep(sleep_time)
4806+ proc_start_time = None
4807+ tries = 0
4808+ while tries <= retry_count and not proc_start_time:
4809+ try:
4810+ proc_start_time = self._get_proc_start_time(sentry_unit,
4811+ service,
4812+ pgrep_full)
4813+ self.log.debug('Attempt {} to get {} proc start time on {} '
4814+ 'OK'.format(tries, service, unit_name))
4815+ except IOError as e:
4816+ # NOTE(beisner) - race avoidance, proc may not exist yet.
4817+ # https://bugs.launchpad.net/charm-helpers/+bug/1474030
4818+ self.log.debug('Attempt {} to get {} proc start time on {} '
4819+ 'failed\n{}'.format(tries, service,
4820+ unit_name, e))
4821+ time.sleep(retry_sleep_time)
4822+ tries += 1
4823+
4824+ if not proc_start_time:
4825+ self.log.warn('No proc start time found, assuming service did '
4826+ 'not start')
4827+ return False
4828+ if proc_start_time >= mtime:
4829+ self.log.debug('Proc start time is newer than provided mtime'
4830+ '(%s >= %s) on %s (OK)' % (proc_start_time,
4831+ mtime, unit_name))
4832+ return True
4833+ else:
4834+ self.log.warn('Proc start time (%s) is older than provided mtime '
4835+ '(%s) on %s, service did not '
4836+ 'restart' % (proc_start_time, mtime, unit_name))
4837+ return False
4838+
4839+ def config_updated_since(self, sentry_unit, filename, mtime,
4840+ sleep_time=20, retry_count=30,
4841+ retry_sleep_time=10):
4842+ """Check if file was modified after a given time.
4843+
4844+ Args:
4845+ sentry_unit (sentry): The sentry unit to check the file mtime on
4846+ filename (string): The file to check mtime of
4847+ mtime (float): The epoch time to check against
4848+ sleep_time (int): Initial sleep time (s) before looking for file
4849+ retry_sleep_time (int): Time (s) to sleep between retries
4850+ retry_count (int): If file is not found, how many times to retry
4851+
4852+ Returns:
4853+ bool: True if file was modified more recently than mtime, False if
4854+ file was modified before mtime, or if file not found.
4855+ """
4856+ unit_name = sentry_unit.info['unit_name']
4857+ self.log.debug('Checking that %s updated since %s on '
4858+ '%s' % (filename, mtime, unit_name))
4859+ time.sleep(sleep_time)
4860+ file_mtime = None
4861+ tries = 0
4862+ while tries <= retry_count and not file_mtime:
4863+ try:
4864+ file_mtime = self._get_file_mtime(sentry_unit, filename)
4865+ self.log.debug('Attempt {} to get {} file mtime on {} '
4866+ 'OK'.format(tries, filename, unit_name))
4867+ except IOError as e:
4868+ # NOTE(beisner) - race avoidance, file may not exist yet.
4869+ # https://bugs.launchpad.net/charm-helpers/+bug/1474030
4870+ self.log.debug('Attempt {} to get {} file mtime on {} '
4871+ 'failed\n{}'.format(tries, filename,
4872+ unit_name, e))
4873+ time.sleep(retry_sleep_time)
4874+ tries += 1
4875+
4876+ if not file_mtime:
4877+ self.log.warn('Could not determine file mtime, assuming '
4878+ 'file does not exist')
4879+ return False
4880+
4881+ if file_mtime >= mtime:
4882+ self.log.debug('File mtime is newer than provided mtime '
4883+ '(%s >= %s) on %s (OK)' % (file_mtime,
4884+ mtime, unit_name))
4885+ return True
4886+ else:
4887+ self.log.warn('File mtime is older than provided mtime'
4888+ '(%s < on %s) on %s' % (file_mtime,
4889+ mtime, unit_name))
4890+ return False
4891+
4892+ def validate_service_config_changed(self, sentry_unit, mtime, service,
4893+ filename, pgrep_full=None,
4894+ sleep_time=20, retry_count=30,
4895+ retry_sleep_time=10):
4896+ """Check service and file were updated after mtime
4897+
4898+ Args:
4899+ sentry_unit (sentry): The sentry unit to check for the service on
4900+ mtime (float): The epoch time to check against
4901+ service (string): service name to look for in process table
4902+ filename (string): The file to check mtime of
4903+ pgrep_full: [Deprecated] Use full command line search mode with pgrep
4904+ sleep_time (int): Initial sleep in seconds to pass to test helpers
4905+ retry_count (int): If service is not found, how many times to retry
4906+ retry_sleep_time (int): Time in seconds to wait between retries
4907+
4908+ Typical Usage:
4909+ u = OpenStackAmuletUtils(ERROR)
4910+ ...
4911+ mtime = u.get_sentry_time(self.cinder_sentry)
4912+ self.d.configure('cinder', {'verbose': 'True', 'debug': 'True'})
4913+ if not u.validate_service_config_changed(self.cinder_sentry,
4914+ mtime,
4915+ 'cinder-api',
4916+ '/etc/cinder/cinder.conf')
4917+ amulet.raise_status(amulet.FAIL, msg='update failed')
4918+ Returns:
4919+ bool: True if both service and file where updated/restarted after
4920+ mtime, False if service is older than mtime or if service was
4921+ not found or if filename was modified before mtime.
4922+ """
4923+
4924+ # NOTE(beisner) pgrep_full is no longer implemented, as pidof is now
4925+ # used instead of pgrep. pgrep_full is still passed through to ensure
4926+ # deprecation WARNS. lp1474030
4927+
4928+ service_restart = self.service_restarted_since(
4929+ sentry_unit, mtime,
4930+ service,
4931+ pgrep_full=pgrep_full,
4932+ sleep_time=sleep_time,
4933+ retry_count=retry_count,
4934+ retry_sleep_time=retry_sleep_time)
4935+
4936+ config_update = self.config_updated_since(
4937+ sentry_unit,
4938+ filename,
4939+ mtime,
4940+ sleep_time=sleep_time,
4941+ retry_count=retry_count,
4942+ retry_sleep_time=retry_sleep_time)
4943+
4944+ return service_restart and config_update
4945+
4946+ def get_sentry_time(self, sentry_unit):
4947+ """Return current epoch time on a sentry"""
4948+ cmd = "date +'%s'"
4949+ return float(sentry_unit.run(cmd)[0])
4950+
4951+ def relation_error(self, name, data):
4952+ return 'unexpected relation data in {} - {}'.format(name, data)
4953+
4954+ def endpoint_error(self, name, data):
4955+ return 'unexpected endpoint data in {} - {}'.format(name, data)
4956+
4957+ def get_ubuntu_releases(self):
4958+ """Return a list of all Ubuntu releases in order of release."""
4959+ _d = distro_info.UbuntuDistroInfo()
4960+ _release_list = _d.all
4961+ return _release_list
4962+
4963+ def file_to_url(self, file_rel_path):
4964+ """Convert a relative file path to a file URL."""
4965+ _abs_path = os.path.abspath(file_rel_path)
4966+ return urlparse.urlparse(_abs_path, scheme='file').geturl()
4967+
4968+ def check_commands_on_units(self, commands, sentry_units):
4969+ """Check that all commands in a list exit zero on all
4970+ sentry units in a list.
4971+
4972+ :param commands: list of bash commands
4973+ :param sentry_units: list of sentry unit pointers
4974+ :returns: None if successful; Failure message otherwise
4975+ """
4976+ self.log.debug('Checking exit codes for {} commands on {} '
4977+ 'sentry units...'.format(len(commands),
4978+ len(sentry_units)))
4979+ for sentry_unit in sentry_units:
4980+ for cmd in commands:
4981+ output, code = sentry_unit.run(cmd)
4982+ if code == 0:
4983+ self.log.debug('{} `{}` returned {} '
4984+ '(OK)'.format(sentry_unit.info['unit_name'],
4985+ cmd, code))
4986+ else:
4987+ return ('{} `{}` returned {} '
4988+ '{}'.format(sentry_unit.info['unit_name'],
4989+ cmd, code, output))
4990+ return None
4991+
4992+ def get_process_id_list(self, sentry_unit, process_name,
4993+ expect_success=True):
4994+ """Get a list of process ID(s) from a single sentry juju unit
4995+ for a single process name.
4996+
4997+ :param sentry_unit: Amulet sentry instance (juju unit)
4998+ :param process_name: Process name
4999+ :param expect_success: If False, expect the PID to be missing,
5000+ raise if it is present.
The diff has been truncated for viewing.

Subscribers

People subscribed via source and target branches

to all changes: