Merge lp:~1chb1n/charms/trusty/glance/next-amulet-debug-and-makefile into lp:~openstack-charmers-archive/charms/trusty/glance/trunk

Proposed by Ryan Beisner
Status: Superseded
Proposed branch: lp:~1chb1n/charms/trusty/glance/next-amulet-debug-and-makefile
Merge into: lp:~openstack-charmers-archive/charms/trusty/glance/trunk
Diff against target: 4234 lines (+2604/-318)
57 files modified
.bzrignore (+1/-0)
Makefile (+2/-3)
README.md (+82/-0)
actions.yaml (+2/-0)
actions/git_reinstall.py (+45/-0)
config.yaml (+22/-0)
hooks/charmhelpers/contrib/charmsupport/nrpe.py (+41/-7)
hooks/charmhelpers/contrib/hahelpers/cluster.py (+5/-1)
hooks/charmhelpers/contrib/network/ip.py (+84/-1)
hooks/charmhelpers/contrib/openstack/amulet/deployment.py (+34/-5)
hooks/charmhelpers/contrib/openstack/context.py (+280/-13)
hooks/charmhelpers/contrib/openstack/files/__init__.py (+18/-0)
hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh (+32/-0)
hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh (+30/-0)
hooks/charmhelpers/contrib/openstack/ip.py (+37/-0)
hooks/charmhelpers/contrib/openstack/neutron.py (+83/-0)
hooks/charmhelpers/contrib/openstack/templates/git.upstart (+17/-0)
hooks/charmhelpers/contrib/openstack/templates/section-keystone-authtoken (+9/-0)
hooks/charmhelpers/contrib/openstack/templates/section-rabbitmq-oslo (+22/-0)
hooks/charmhelpers/contrib/openstack/templates/section-zeromq (+14/-0)
hooks/charmhelpers/contrib/openstack/utils.py (+142/-141)
hooks/charmhelpers/contrib/python/packages.py (+2/-2)
hooks/charmhelpers/core/fstab.py (+4/-4)
hooks/charmhelpers/core/hookenv.py (+40/-1)
hooks/charmhelpers/core/host.py (+10/-6)
hooks/charmhelpers/core/services/helpers.py (+12/-4)
hooks/charmhelpers/core/strutils.py (+42/-0)
hooks/charmhelpers/core/sysctl.py (+2/-2)
hooks/charmhelpers/core/templating.py (+3/-3)
hooks/charmhelpers/core/unitdata.py (+477/-0)
hooks/charmhelpers/fetch/archiveurl.py (+10/-10)
hooks/charmhelpers/fetch/giturl.py (+1/-1)
hooks/glance_relations.py (+43/-17)
hooks/glance_utils.py (+139/-9)
templates/kilo/glance-api-paste.ini (+77/-0)
templates/kilo/glance-api.conf (+83/-0)
templates/kilo/glance-registry-paste.ini (+30/-0)
templates/kilo/glance-registry.conf (+27/-0)
templates/parts/keystone (+1/-0)
templates/parts/section-database (+1/-0)
tests/016-basic-trusty-juno (+11/-0)
tests/017-basic-trusty-kilo (+11/-0)
tests/018-basic-utopic-juno (+9/-0)
tests/019-basic-vivid-kilo (+9/-0)
tests/050-basic-trusty-icehouse-git (+9/-0)
tests/051-basic-trusty-juno-git (+12/-0)
tests/10-basic-precise-essex (+0/-9)
tests/11-basic-precise-folsom (+0/-11)
tests/12-basic-precise-grizzly (+0/-11)
tests/13-basic-precise-havana (+0/-11)
tests/basic_deployment.py (+27/-4)
tests/charmhelpers/contrib/amulet/utils.py (+125/-3)
tests/charmhelpers/contrib/openstack/amulet/deployment.py (+34/-5)
unit_tests/__init__.py (+1/-0)
unit_tests/test_actions_git_reinstall.py (+96/-0)
unit_tests/test_glance_relations.py (+113/-26)
unit_tests/test_glance_utils.py (+141/-8)
To merge this branch: bzr merge lp:~1chb1n/charms/trusty/glance/next-amulet-debug-and-makefile
Reviewer Review Type Date Requested Status
OpenStack Charmers Pending
Review via email: mp+256581@code.launchpad.net

This proposal has been superseded by a proposal from 2015-04-17.

Description of the change

auto normalize amulet test definitions and amulet make targets; charm-helper sync.

To post a comment you must log in.
Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #3538 glance for 1chb1n mp256581
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3538/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #3326 glance for 1chb1n mp256581
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3326/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #3293 glance for 1chb1n mp256581
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/10835861/
Build: http://10.245.162.77:8080/job/charm_amulet_test/3293/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_lint_check #3577 glance for 1chb1n mp256581
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/3577/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_unit_test #3365 glance for 1chb1n mp256581
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/3365/

Revision history for this message
uosci-testing-bot (uosci-testing-bot) wrote :

charm_amulet_test #3334 glance for 1chb1n mp256581
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/3334/

Unmerged revisions

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file '.bzrignore'
2--- .bzrignore 2014-07-02 08:09:03 +0000
3+++ .bzrignore 2015-04-16 21:50:07 +0000
4@@ -1,2 +1,3 @@
5 .coverage
6 bin
7+tags
8
9=== modified file 'Makefile'
10--- Makefile 2014-10-08 20:18:38 +0000
11+++ Makefile 2015-04-16 21:50:07 +0000
12@@ -3,7 +3,7 @@
13
14 lint:
15 @echo "Running flake8 tests: "
16- @flake8 --exclude hooks/charmhelpers hooks unit_tests tests
17+ @flake8 --exclude hooks/charmhelpers actions hooks unit_tests tests
18 @echo "OK"
19 @echo "Running charm proof: "
20 @charm proof
21@@ -26,8 +26,7 @@
22 # /!\ Note: The -v should only be temporary until Amulet sends
23 # raise_status() messages to stderr:
24 # https://bugs.launchpad.net/amulet/+bug/1320357
25- @juju test -v -p AMULET_HTTP_PROXY --timeout 900 \
26- 00-setup 14-basic-precise-icehouse 15-basic-trusty-icehouse
27+ @juju test -v -p AMULET_HTTP_PROXY,AMULET_OS_VIP --timeout 2700
28
29 publish: lint unit_test
30 bzr push lp:charms/glance
31
32=== modified file 'README.md'
33--- README.md 2013-09-26 10:22:09 +0000
34+++ README.md 2015-04-16 21:50:07 +0000
35@@ -81,6 +81,88 @@
36 Note that Glance in this configuration must be used with either Ceph or
37 Swift providing backing image storage.
38
39+Deploying from source
40+---------------------
41+
42+The minimum openstack-origin-git config required to deploy from source is:
43+
44+ openstack-origin-git:
45+ "repositories:
46+ - {name: requirements,
47+ repository: 'git://git.openstack.org/openstack/requirements',
48+ branch: stable/juno}
49+ - {name: glance,
50+ repository: 'git://git.openstack.org/openstack/glance',
51+ branch: stable/juno}"
52+
53+Note that there are only two 'name' values the charm knows about: 'requirements'
54+and 'glance'. These repositories must correspond to these 'name' values.
55+Additionally, the requirements repository must be specified first and the
56+glance repository must be specified last. All other repostories are installed
57+in the order in which they are specified.
58+
59+The following is a full list of current tip repos (may not be up-to-date):
60+
61+ openstack-origin-git:
62+ "repositories:
63+ - {name: requirements,
64+ repository: 'git://git.openstack.org/openstack/requirements',
65+ branch: master}
66+ - {name: oslo-concurrency,
67+ repository: 'git://git.openstack.org/openstack/oslo.concurrency',
68+ branch: master}
69+ - {name: oslo-config,
70+ repository: 'git://git.openstack.org/openstack/oslo.config',
71+ branch: master}
72+ - {name: oslo-db,
73+ repository: 'git://git.openstack.org/openstack/oslo.db',
74+ branch: master}
75+ - {name: oslo-i18n,
76+ repository: 'git://git.openstack.org/openstack/oslo.i18n',
77+ branch: master}
78+ - {name: oslo-messaging,
79+ repository: 'git://git.openstack.org/openstack/oslo.messaging',
80+ branch: master}
81+ - {name: oslo-serialization,
82+ repository: 'git://git.openstack.org/openstack/oslo.serialization',
83+ branch: master}
84+ - {name: oslo-utils,
85+ repository: 'git://git.openstack.org/openstack/oslo.utils',
86+ branch: master}
87+ - {name: oslo-vmware,
88+ repository: 'git://git.openstack.org/openstack/oslo.vmware',
89+ branch: master}
90+ - {name: osprofiler,
91+ repository: 'git://git.openstack.org/stackforge/osprofiler',
92+ branch: master}
93+ - {name: pbr,
94+ repository: 'git://git.openstack.org/openstack-dev/pbr',
95+ branch: master}
96+ - {name: python-keystoneclient,
97+ repository: 'git://git.openstack.org/openstack/python-keystoneclient',
98+ branch: master}
99+ - {name: python-swiftclient,
100+ repository: 'git://git.openstack.org/openstack/python-swiftclient',
101+ branch: master}
102+ - {name: sqlalchemy-migrate,
103+ repository: 'git://git.openstack.org/stackforge/sqlalchemy-migrate',
104+ branch: master}
105+ - {name: stevedore,
106+ repository: 'git://git.openstack.org/openstack/stevedore',
107+ branch: master}
108+ - {name: wsme,
109+ repository: 'git://git.openstack.org/stackforge/wsme',
110+ branch: master}
111+ - {name: keystonemiddleware,
112+ repository: 'git://git.openstack.org/openstack/keystonemiddleware',
113+ branch: master}
114+ - {name: glance-store,
115+ repository: 'git://git.openstack.org/openstack/glance_store',
116+ branch: master}
117+ - {name: glance,
118+ repository: 'git://git.openstack.org/openstack/glance',
119+ branch: master}"
120+
121 Contact Information
122 -------------------
123
124
125=== added directory 'actions'
126=== added file 'actions.yaml'
127--- actions.yaml 1970-01-01 00:00:00 +0000
128+++ actions.yaml 2015-04-16 21:50:07 +0000
129@@ -0,0 +1,2 @@
130+git-reinstall:
131+ description: Reinstall glance from the openstack-origin-git repositories.
132
133=== added symlink 'actions/git-reinstall'
134=== target is u'git_reinstall.py'
135=== added file 'actions/git_reinstall.py'
136--- actions/git_reinstall.py 1970-01-01 00:00:00 +0000
137+++ actions/git_reinstall.py 2015-04-16 21:50:07 +0000
138@@ -0,0 +1,45 @@
139+#!/usr/bin/python
140+import sys
141+import traceback
142+
143+sys.path.append('hooks/')
144+
145+from charmhelpers.contrib.openstack.utils import (
146+ git_install_requested,
147+)
148+
149+from charmhelpers.core.hookenv import (
150+ action_set,
151+ action_fail,
152+ config,
153+)
154+
155+from glance_utils import (
156+ git_install,
157+)
158+
159+from glance_relations import (
160+ config_changed,
161+)
162+
163+
164+def git_reinstall():
165+ """Reinstall from source and restart services.
166+
167+ If the openstack-origin-git config option was used to install openstack
168+ from source git repositories, then this action can be used to reinstall
169+ from updated git repositories, followed by a restart of services."""
170+ if not git_install_requested():
171+ action_fail('openstack-origin-git is not configured')
172+ return
173+
174+ try:
175+ git_install(config('openstack-origin-git'))
176+ config_changed()
177+ except:
178+ action_set({'traceback': traceback.format_exc()})
179+ action_fail('git-reinstall resulted in an unexpected error')
180+
181+
182+if __name__ == '__main__':
183+ git_reinstall()
184
185=== modified file 'config.yaml'
186--- config.yaml 2015-01-21 14:38:50 +0000
187+++ config.yaml 2015-04-16 21:50:07 +0000
188@@ -14,6 +14,22 @@
189 Note that updating this setting to a source that is known to
190 provide a later version of OpenStack will trigger a software
191 upgrade.
192+
193+ Note that when openstack-origin-git is specified, openstack
194+ specific packages will be installed from source rather than
195+ from the openstack-origin repository.
196+ openstack-origin-git:
197+ default:
198+ type: string
199+ description: |
200+ Specifies a YAML-formatted dictionary listing the git
201+ repositories and branches from which to install OpenStack and
202+ its dependencies.
203+
204+ Note that the installed config files will be determined based on
205+ the OpenStack release of the openstack-origin option.
206+
207+ For more details see README.md.
208 database-user:
209 default: glance
210 type: string
211@@ -189,4 +205,10 @@
212 juju-myservice-0
213 If you're running multiple environments with the same services in them
214 this allows you to differentiate between them.
215+ nagios_servicegroups:
216+ default: ""
217+ type: string
218+ description: |
219+ A comma-separated list of nagios servicegroups.
220+ If left empty, the nagios_context will be used as the servicegroup
221
222
223=== modified file 'hooks/charmhelpers/contrib/charmsupport/nrpe.py'
224--- hooks/charmhelpers/contrib/charmsupport/nrpe.py 2015-01-26 09:45:23 +0000
225+++ hooks/charmhelpers/contrib/charmsupport/nrpe.py 2015-04-16 21:50:07 +0000
226@@ -24,6 +24,8 @@
227 import pwd
228 import grp
229 import os
230+import glob
231+import shutil
232 import re
233 import shlex
234 import yaml
235@@ -161,7 +163,7 @@
236 log('Check command not found: {}'.format(parts[0]))
237 return ''
238
239- def write(self, nagios_context, hostname, nagios_servicegroups=None):
240+ def write(self, nagios_context, hostname, nagios_servicegroups):
241 nrpe_check_file = '/etc/nagios/nrpe.d/{}.cfg'.format(
242 self.command)
243 with open(nrpe_check_file, 'w') as nrpe_check_config:
244@@ -177,14 +179,11 @@
245 nagios_servicegroups)
246
247 def write_service_config(self, nagios_context, hostname,
248- nagios_servicegroups=None):
249+ nagios_servicegroups):
250 for f in os.listdir(NRPE.nagios_exportdir):
251 if re.search('.*{}.cfg'.format(self.command), f):
252 os.remove(os.path.join(NRPE.nagios_exportdir, f))
253
254- if not nagios_servicegroups:
255- nagios_servicegroups = nagios_context
256-
257 templ_vars = {
258 'nagios_hostname': hostname,
259 'nagios_servicegroup': nagios_servicegroups,
260@@ -211,10 +210,10 @@
261 super(NRPE, self).__init__()
262 self.config = config()
263 self.nagios_context = self.config['nagios_context']
264- if 'nagios_servicegroups' in self.config:
265+ if 'nagios_servicegroups' in self.config and self.config['nagios_servicegroups']:
266 self.nagios_servicegroups = self.config['nagios_servicegroups']
267 else:
268- self.nagios_servicegroups = 'juju'
269+ self.nagios_servicegroups = self.nagios_context
270 self.unit_name = local_unit().replace('/', '-')
271 if hostname:
272 self.hostname = hostname
273@@ -322,3 +321,38 @@
274 check_cmd='check_status_file.py -f '
275 '/var/lib/nagios/service-check-%s.txt' % svc,
276 )
277+
278+
279+def copy_nrpe_checks():
280+ """
281+ Copy the nrpe checks into place
282+
283+ """
284+ NAGIOS_PLUGINS = '/usr/local/lib/nagios/plugins'
285+ nrpe_files_dir = os.path.join(os.getenv('CHARM_DIR'), 'hooks',
286+ 'charmhelpers', 'contrib', 'openstack',
287+ 'files')
288+
289+ if not os.path.exists(NAGIOS_PLUGINS):
290+ os.makedirs(NAGIOS_PLUGINS)
291+ for fname in glob.glob(os.path.join(nrpe_files_dir, "check_*")):
292+ if os.path.isfile(fname):
293+ shutil.copy2(fname,
294+ os.path.join(NAGIOS_PLUGINS, os.path.basename(fname)))
295+
296+
297+def add_haproxy_checks(nrpe, unit_name):
298+ """
299+ Add checks for each service in list
300+
301+ :param NRPE nrpe: NRPE object to add check to
302+ :param str unit_name: Unit name to use in check description
303+ """
304+ nrpe.add_check(
305+ shortname='haproxy_servers',
306+ description='Check HAProxy {%s}' % unit_name,
307+ check_cmd='check_haproxy.sh')
308+ nrpe.add_check(
309+ shortname='haproxy_queue',
310+ description='Check HAProxy queue depth {%s}' % unit_name,
311+ check_cmd='check_haproxy_queue_depth.sh')
312
313=== modified file 'hooks/charmhelpers/contrib/hahelpers/cluster.py'
314--- hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-01-26 09:45:23 +0000
315+++ hooks/charmhelpers/contrib/hahelpers/cluster.py 2015-04-16 21:50:07 +0000
316@@ -48,6 +48,9 @@
317 from charmhelpers.core.decorators import (
318 retry_on_exception,
319 )
320+from charmhelpers.core.strutils import (
321+ bool_from_string,
322+)
323
324
325 class HAIncompleteConfig(Exception):
326@@ -164,7 +167,8 @@
327 .
328 returns: boolean
329 '''
330- if config_get('use-https') == "yes":
331+ use_https = config_get('use-https')
332+ if use_https and bool_from_string(use_https):
333 return True
334 if config_get('ssl_cert') and config_get('ssl_key'):
335 return True
336
337=== modified file 'hooks/charmhelpers/contrib/network/ip.py'
338--- hooks/charmhelpers/contrib/network/ip.py 2015-01-26 09:45:23 +0000
339+++ hooks/charmhelpers/contrib/network/ip.py 2015-04-16 21:50:07 +0000
340@@ -17,13 +17,16 @@
341 import glob
342 import re
343 import subprocess
344+import six
345+import socket
346
347 from functools import partial
348
349 from charmhelpers.core.hookenv import unit_get
350 from charmhelpers.fetch import apt_install
351 from charmhelpers.core.hookenv import (
352- log
353+ log,
354+ WARNING,
355 )
356
357 try:
358@@ -365,3 +368,83 @@
359 return True
360
361 return False
362+
363+
364+def is_ip(address):
365+ """
366+ Returns True if address is a valid IP address.
367+ """
368+ try:
369+ # Test to see if already an IPv4 address
370+ socket.inet_aton(address)
371+ return True
372+ except socket.error:
373+ return False
374+
375+
376+def ns_query(address):
377+ try:
378+ import dns.resolver
379+ except ImportError:
380+ apt_install('python-dnspython')
381+ import dns.resolver
382+
383+ if isinstance(address, dns.name.Name):
384+ rtype = 'PTR'
385+ elif isinstance(address, six.string_types):
386+ rtype = 'A'
387+ else:
388+ return None
389+
390+ answers = dns.resolver.query(address, rtype)
391+ if answers:
392+ return str(answers[0])
393+ return None
394+
395+
396+def get_host_ip(hostname, fallback=None):
397+ """
398+ Resolves the IP for a given hostname, or returns
399+ the input if it is already an IP.
400+ """
401+ if is_ip(hostname):
402+ return hostname
403+
404+ ip_addr = ns_query(hostname)
405+ if not ip_addr:
406+ try:
407+ ip_addr = socket.gethostbyname(hostname)
408+ except:
409+ log("Failed to resolve hostname '%s'" % (hostname),
410+ level=WARNING)
411+ return fallback
412+ return ip_addr
413+
414+
415+def get_hostname(address, fqdn=True):
416+ """
417+ Resolves hostname for given IP, or returns the input
418+ if it is already a hostname.
419+ """
420+ if is_ip(address):
421+ try:
422+ import dns.reversename
423+ except ImportError:
424+ apt_install("python-dnspython")
425+ import dns.reversename
426+
427+ rev = dns.reversename.from_address(address)
428+ result = ns_query(rev)
429+ if not result:
430+ return None
431+ else:
432+ result = address
433+
434+ if fqdn:
435+ # strip trailing .
436+ if result.endswith('.'):
437+ return result[:-1]
438+ else:
439+ return result
440+ else:
441+ return result.split('.')[0]
442
443=== modified file 'hooks/charmhelpers/contrib/openstack/amulet/deployment.py'
444--- hooks/charmhelpers/contrib/openstack/amulet/deployment.py 2015-01-26 09:45:23 +0000
445+++ hooks/charmhelpers/contrib/openstack/amulet/deployment.py 2015-04-16 21:50:07 +0000
446@@ -15,6 +15,7 @@
447 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
448
449 import six
450+from collections import OrderedDict
451 from charmhelpers.contrib.amulet.deployment import (
452 AmuletDeployment
453 )
454@@ -43,7 +44,7 @@
455 Determine if the local branch being tested is derived from its
456 stable or next (dev) branch, and based on this, use the corresonding
457 stable or next branches for the other_services."""
458- base_charms = ['mysql', 'mongodb', 'rabbitmq-server']
459+ base_charms = ['mysql', 'mongodb']
460
461 if self.stable:
462 for svc in other_services:
463@@ -71,16 +72,19 @@
464 services.append(this_service)
465 use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph',
466 'ceph-osd', 'ceph-radosgw']
467+ # Openstack subordinate charms do not expose an origin option as that
468+ # is controlled by the principle
469+ ignore = ['neutron-openvswitch']
470
471 if self.openstack:
472 for svc in services:
473- if svc['name'] not in use_source:
474+ if svc['name'] not in use_source + ignore:
475 config = {'openstack-origin': self.openstack}
476 self.d.configure(svc['name'], config)
477
478 if self.source:
479 for svc in services:
480- if svc['name'] in use_source:
481+ if svc['name'] in use_source and svc['name'] not in ignore:
482 config = {'source': self.source}
483 self.d.configure(svc['name'], config)
484
485@@ -97,12 +101,37 @@
486 """
487 (self.precise_essex, self.precise_folsom, self.precise_grizzly,
488 self.precise_havana, self.precise_icehouse,
489- self.trusty_icehouse) = range(6)
490+ self.trusty_icehouse, self.trusty_juno, self.trusty_kilo,
491+ self.utopic_juno, self.vivid_kilo) = range(10)
492 releases = {
493 ('precise', None): self.precise_essex,
494 ('precise', 'cloud:precise-folsom'): self.precise_folsom,
495 ('precise', 'cloud:precise-grizzly'): self.precise_grizzly,
496 ('precise', 'cloud:precise-havana'): self.precise_havana,
497 ('precise', 'cloud:precise-icehouse'): self.precise_icehouse,
498- ('trusty', None): self.trusty_icehouse}
499+ ('trusty', None): self.trusty_icehouse,
500+ ('trusty', 'cloud:trusty-juno'): self.trusty_juno,
501+ ('trusty', 'cloud:trusty-kilo'): self.trusty_kilo,
502+ ('utopic', None): self.utopic_juno,
503+ ('vivid', None): self.vivid_kilo}
504 return releases[(self.series, self.openstack)]
505+
506+ def _get_openstack_release_string(self):
507+ """Get openstack release string.
508+
509+ Return a string representing the openstack release.
510+ """
511+ releases = OrderedDict([
512+ ('precise', 'essex'),
513+ ('quantal', 'folsom'),
514+ ('raring', 'grizzly'),
515+ ('saucy', 'havana'),
516+ ('trusty', 'icehouse'),
517+ ('utopic', 'juno'),
518+ ('vivid', 'kilo'),
519+ ])
520+ if self.openstack:
521+ os_origin = self.openstack.split(':')[1]
522+ return os_origin.split('%s-' % self.series)[1].split('/')[0]
523+ else:
524+ return releases[self.series]
525
526=== modified file 'hooks/charmhelpers/contrib/openstack/context.py'
527--- hooks/charmhelpers/contrib/openstack/context.py 2015-03-19 21:56:40 +0000
528+++ hooks/charmhelpers/contrib/openstack/context.py 2015-04-16 21:50:07 +0000
529@@ -16,11 +16,13 @@
530
531 import json
532 import os
533+import re
534 import time
535 from base64 import b64decode
536 from subprocess import check_call
537
538 import six
539+import yaml
540
541 from charmhelpers.fetch import (
542 apt_install,
543@@ -45,8 +47,11 @@
544 )
545
546 from charmhelpers.core.sysctl import create as sysctl_create
547+from charmhelpers.core.strutils import bool_from_string
548
549 from charmhelpers.core.host import (
550+ list_nics,
551+ get_nic_hwaddr,
552 mkdir,
553 write_file,
554 )
555@@ -63,6 +68,11 @@
556 )
557 from charmhelpers.contrib.openstack.neutron import (
558 neutron_plugin_attribute,
559+ parse_data_port_mappings,
560+)
561+from charmhelpers.contrib.openstack.ip import (
562+ resolve_address,
563+ INTERNAL,
564 )
565 from charmhelpers.contrib.openstack.ip import (
566 resolve_address,
567@@ -70,13 +80,14 @@
568 )
569 from charmhelpers.contrib.network.ip import (
570 get_address_in_network,
571+ get_ipv4_addr,
572 get_ipv6_addr,
573 get_netmask_for_address,
574 format_ipv6_addr,
575 is_address_in_network,
576+ is_bridge_member,
577 )
578 from charmhelpers.contrib.openstack.utils import get_host_ip
579-
580 CA_CERT_PATH = '/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt'
581 ADDRESS_TYPES = ['admin', 'internal', 'public']
582
583@@ -108,9 +119,41 @@
584 def config_flags_parser(config_flags):
585 """Parses config flags string into dict.
586
587+ This parsing method supports a few different formats for the config
588+ flag values to be parsed:
589+
590+ 1. A string in the simple format of key=value pairs, with the possibility
591+ of specifying multiple key value pairs within the same string. For
592+ example, a string in the format of 'key1=value1, key2=value2' will
593+ return a dict of:
594+ {'key1': 'value1',
595+ 'key2': 'value2'}.
596+
597+ 2. A string in the above format, but supporting a comma-delimited list
598+ of values for the same key. For example, a string in the format of
599+ 'key1=value1, key2=value3,value4,value5' will return a dict of:
600+ {'key1', 'value1',
601+ 'key2', 'value2,value3,value4'}
602+
603+ 3. A string containing a colon character (:) prior to an equal
604+ character (=) will be treated as yaml and parsed as such. This can be
605+ used to specify more complex key value pairs. For example,
606+ a string in the format of 'key1: subkey1=value1, subkey2=value2' will
607+ return a dict of:
608+ {'key1', 'subkey1=value1, subkey2=value2'}
609+
610 The provided config_flags string may be a list of comma-separated values
611 which themselves may be comma-separated list of values.
612 """
613+ # If we find a colon before an equals sign then treat it as yaml.
614+ # Note: limit it to finding the colon first since this indicates assignment
615+ # for inline yaml.
616+ colon = config_flags.find(':')
617+ equals = config_flags.find('=')
618+ if colon > 0:
619+ if colon < equals or equals < 0:
620+ return yaml.safe_load(config_flags)
621+
622 if config_flags.find('==') >= 0:
623 log("config_flags is not in expected format (key=value)", level=ERROR)
624 raise OSContextError
625@@ -281,12 +324,29 @@
626
627
628 class IdentityServiceContext(OSContextGenerator):
629- interfaces = ['identity-service']
630+
631+ def __init__(self, service=None, service_user=None, rel_name='identity-service'):
632+ self.service = service
633+ self.service_user = service_user
634+ self.rel_name = rel_name
635+ self.interfaces = [self.rel_name]
636
637 def __call__(self):
638- log('Generating template context for identity-service', level=DEBUG)
639+ log('Generating template context for ' + self.rel_name, level=DEBUG)
640 ctxt = {}
641- for rid in relation_ids('identity-service'):
642+
643+ if self.service and self.service_user:
644+ # This is required for pki token signing if we don't want /tmp to
645+ # be used.
646+ cachedir = '/var/cache/%s' % (self.service)
647+ if not os.path.isdir(cachedir):
648+ log("Creating service cache dir %s" % (cachedir), level=DEBUG)
649+ mkdir(path=cachedir, owner=self.service_user,
650+ group=self.service_user, perms=0o700)
651+
652+ ctxt['signing_dir'] = cachedir
653+
654+ for rid in relation_ids(self.rel_name):
655 for unit in related_units(rid):
656 rdata = relation_get(rid=rid, unit=unit)
657 serv_host = rdata.get('service_host')
658@@ -295,15 +355,16 @@
659 auth_host = format_ipv6_addr(auth_host) or auth_host
660 svc_protocol = rdata.get('service_protocol') or 'http'
661 auth_protocol = rdata.get('auth_protocol') or 'http'
662- ctxt = {'service_port': rdata.get('service_port'),
663- 'service_host': serv_host,
664- 'auth_host': auth_host,
665- 'auth_port': rdata.get('auth_port'),
666- 'admin_tenant_name': rdata.get('service_tenant'),
667- 'admin_user': rdata.get('service_username'),
668- 'admin_password': rdata.get('service_password'),
669- 'service_protocol': svc_protocol,
670- 'auth_protocol': auth_protocol}
671+ ctxt.update({'service_port': rdata.get('service_port'),
672+ 'service_host': serv_host,
673+ 'auth_host': auth_host,
674+ 'auth_port': rdata.get('auth_port'),
675+ 'admin_tenant_name': rdata.get('service_tenant'),
676+ 'admin_user': rdata.get('service_username'),
677+ 'admin_password': rdata.get('service_password'),
678+ 'service_protocol': svc_protocol,
679+ 'auth_protocol': auth_protocol})
680+
681 if context_complete(ctxt):
682 # NOTE(jamespage) this is required for >= icehouse
683 # so a missing value just indicates keystone needs
684@@ -402,6 +463,11 @@
685
686 ctxt['rabbitmq_hosts'] = ','.join(sorted(rabbitmq_hosts))
687
688+ oslo_messaging_flags = conf.get('oslo-messaging-flags', None)
689+ if oslo_messaging_flags:
690+ ctxt['oslo_messaging_flags'] = config_flags_parser(
691+ oslo_messaging_flags)
692+
693 if not context_complete(ctxt):
694 return {}
695
696@@ -751,6 +817,19 @@
697
698 return ovs_ctxt
699
700+ def nuage_ctxt(self):
701+ driver = neutron_plugin_attribute(self.plugin, 'driver',
702+ self.network_manager)
703+ config = neutron_plugin_attribute(self.plugin, 'config',
704+ self.network_manager)
705+ nuage_ctxt = {'core_plugin': driver,
706+ 'neutron_plugin': 'vsp',
707+ 'neutron_security_groups': self.neutron_security_groups,
708+ 'local_ip': unit_private_ip(),
709+ 'config': config}
710+
711+ return nuage_ctxt
712+
713 def nvp_ctxt(self):
714 driver = neutron_plugin_attribute(self.plugin, 'driver',
715 self.network_manager)
716@@ -834,6 +913,8 @@
717 ctxt.update(self.n1kv_ctxt())
718 elif self.plugin == 'Calico':
719 ctxt.update(self.calico_ctxt())
720+ elif self.plugin == 'vsp':
721+ ctxt.update(self.nuage_ctxt())
722
723 alchemy_flags = config('neutron-alchemy-flags')
724 if alchemy_flags:
725@@ -844,6 +925,48 @@
726 return ctxt
727
728
729+class NeutronPortContext(OSContextGenerator):
730+ NIC_PREFIXES = ['eth', 'bond']
731+
732+ def resolve_ports(self, ports):
733+ """Resolve NICs not yet bound to bridge(s)
734+
735+ If hwaddress provided then returns resolved hwaddress otherwise NIC.
736+ """
737+ if not ports:
738+ return None
739+
740+ hwaddr_to_nic = {}
741+ hwaddr_to_ip = {}
742+ for nic in list_nics(self.NIC_PREFIXES):
743+ hwaddr = get_nic_hwaddr(nic)
744+ hwaddr_to_nic[hwaddr] = nic
745+ addresses = get_ipv4_addr(nic, fatal=False)
746+ addresses += get_ipv6_addr(iface=nic, fatal=False)
747+ hwaddr_to_ip[hwaddr] = addresses
748+
749+ resolved = []
750+ mac_regex = re.compile(r'([0-9A-F]{2}[:-]){5}([0-9A-F]{2})', re.I)
751+ for entry in ports:
752+ if re.match(mac_regex, entry):
753+ # NIC is in known NICs and does NOT hace an IP address
754+ if entry in hwaddr_to_nic and not hwaddr_to_ip[entry]:
755+ # If the nic is part of a bridge then don't use it
756+ if is_bridge_member(hwaddr_to_nic[entry]):
757+ continue
758+
759+ # Entry is a MAC address for a valid interface that doesn't
760+ # have an IP address assigned yet.
761+ resolved.append(hwaddr_to_nic[entry])
762+ else:
763+ # If the passed entry is not a MAC address, assume it's a valid
764+ # interface, and that the user put it there on purpose (we can
765+ # trust it to be the real external network).
766+ resolved.append(entry)
767+
768+ return resolved
769+
770+
771 class OSConfigFlagContext(OSContextGenerator):
772 """Provides support for user-defined config flags.
773
774@@ -1032,6 +1155,8 @@
775 for unit in related_units(rid):
776 ctxt['zmq_nonce'] = relation_get('nonce', unit, rid)
777 ctxt['zmq_host'] = relation_get('host', unit, rid)
778+ ctxt['zmq_redis_address'] = relation_get(
779+ 'zmq_redis_address', unit, rid)
780
781 return ctxt
782
783@@ -1063,3 +1188,145 @@
784 sysctl_create(sysctl_dict,
785 '/etc/sysctl.d/50-{0}.conf'.format(charm_name()))
786 return {'sysctl': sysctl_dict}
787+
788+
789+class NeutronAPIContext(OSContextGenerator):
790+ '''
791+ Inspects current neutron-plugin-api relation for neutron settings. Return
792+ defaults if it is not present.
793+ '''
794+ interfaces = ['neutron-plugin-api']
795+
796+ def __call__(self):
797+ self.neutron_defaults = {
798+ 'l2_population': {
799+ 'rel_key': 'l2-population',
800+ 'default': False,
801+ },
802+ 'overlay_network_type': {
803+ 'rel_key': 'overlay-network-type',
804+ 'default': 'gre',
805+ },
806+ 'neutron_security_groups': {
807+ 'rel_key': 'neutron-security-groups',
808+ 'default': False,
809+ },
810+ 'network_device_mtu': {
811+ 'rel_key': 'network-device-mtu',
812+ 'default': None,
813+ },
814+ 'enable_dvr': {
815+ 'rel_key': 'enable-dvr',
816+ 'default': False,
817+ },
818+ 'enable_l3ha': {
819+ 'rel_key': 'enable-l3ha',
820+ 'default': False,
821+ },
822+ }
823+ ctxt = self.get_neutron_options({})
824+ for rid in relation_ids('neutron-plugin-api'):
825+ for unit in related_units(rid):
826+ rdata = relation_get(rid=rid, unit=unit)
827+ if 'l2-population' in rdata:
828+ ctxt.update(self.get_neutron_options(rdata))
829+
830+ return ctxt
831+
832+ def get_neutron_options(self, rdata):
833+ settings = {}
834+ for nkey in self.neutron_defaults.keys():
835+ defv = self.neutron_defaults[nkey]['default']
836+ rkey = self.neutron_defaults[nkey]['rel_key']
837+ if rkey in rdata.keys():
838+ if type(defv) is bool:
839+ settings[nkey] = bool_from_string(rdata[rkey])
840+ else:
841+ settings[nkey] = rdata[rkey]
842+ else:
843+ settings[nkey] = defv
844+ return settings
845+
846+
847+class ExternalPortContext(NeutronPortContext):
848+
849+ def __call__(self):
850+ ctxt = {}
851+ ports = config('ext-port')
852+ if ports:
853+ ports = [p.strip() for p in ports.split()]
854+ ports = self.resolve_ports(ports)
855+ if ports:
856+ ctxt = {"ext_port": ports[0]}
857+ napi_settings = NeutronAPIContext()()
858+ mtu = napi_settings.get('network_device_mtu')
859+ if mtu:
860+ ctxt['ext_port_mtu'] = mtu
861+
862+ return ctxt
863+
864+
865+class DataPortContext(NeutronPortContext):
866+
867+ def __call__(self):
868+ ports = config('data-port')
869+ if ports:
870+ portmap = parse_data_port_mappings(ports)
871+ ports = portmap.values()
872+ resolved = self.resolve_ports(ports)
873+ normalized = {get_nic_hwaddr(port): port for port in resolved
874+ if port not in ports}
875+ normalized.update({port: port for port in resolved
876+ if port in ports})
877+ if resolved:
878+ return {bridge: normalized[port] for bridge, port in
879+ six.iteritems(portmap) if port in normalized.keys()}
880+
881+ return None
882+
883+
884+class PhyNICMTUContext(DataPortContext):
885+
886+ def __call__(self):
887+ ctxt = {}
888+ mappings = super(PhyNICMTUContext, self).__call__()
889+ if mappings and mappings.values():
890+ ports = mappings.values()
891+ napi_settings = NeutronAPIContext()()
892+ mtu = napi_settings.get('network_device_mtu')
893+ if mtu:
894+ ctxt["devs"] = '\\n'.join(ports)
895+ ctxt['mtu'] = mtu
896+
897+ return ctxt
898+
899+
900+class NetworkServiceContext(OSContextGenerator):
901+
902+ def __init__(self, rel_name='quantum-network-service'):
903+ self.rel_name = rel_name
904+ self.interfaces = [rel_name]
905+
906+ def __call__(self):
907+ for rid in relation_ids(self.rel_name):
908+ for unit in related_units(rid):
909+ rdata = relation_get(rid=rid, unit=unit)
910+ ctxt = {
911+ 'keystone_host': rdata.get('keystone_host'),
912+ 'service_port': rdata.get('service_port'),
913+ 'auth_port': rdata.get('auth_port'),
914+ 'service_tenant': rdata.get('service_tenant'),
915+ 'service_username': rdata.get('service_username'),
916+ 'service_password': rdata.get('service_password'),
917+ 'quantum_host': rdata.get('quantum_host'),
918+ 'quantum_port': rdata.get('quantum_port'),
919+ 'quantum_url': rdata.get('quantum_url'),
920+ 'region': rdata.get('region'),
921+ 'service_protocol':
922+ rdata.get('service_protocol') or 'http',
923+ 'auth_protocol':
924+ rdata.get('auth_protocol') or 'http',
925+ }
926+ if context_complete(ctxt):
927+ return ctxt
928+ return {}
929
930=== added directory 'hooks/charmhelpers/contrib/openstack/files'
931=== added file 'hooks/charmhelpers/contrib/openstack/files/__init__.py'
932--- hooks/charmhelpers/contrib/openstack/files/__init__.py 1970-01-01 00:00:00 +0000
933+++ hooks/charmhelpers/contrib/openstack/files/__init__.py 2015-04-16 21:50:07 +0000
934@@ -0,0 +1,18 @@
935+# Copyright 2014-2015 Canonical Limited.
936+#
937+# This file is part of charm-helpers.
938+#
939+# charm-helpers is free software: you can redistribute it and/or modify
940+# it under the terms of the GNU Lesser General Public License version 3 as
941+# published by the Free Software Foundation.
942+#
943+# charm-helpers is distributed in the hope that it will be useful,
944+# but WITHOUT ANY WARRANTY; without even the implied warranty of
945+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
946+# GNU Lesser General Public License for more details.
947+#
948+# You should have received a copy of the GNU Lesser General Public License
949+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
950+
951+# dummy __init__.py to fool syncer into thinking this is a syncable python
952+# module
953
954=== added file 'hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh'
955--- hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh 1970-01-01 00:00:00 +0000
956+++ hooks/charmhelpers/contrib/openstack/files/check_haproxy.sh 2015-04-16 21:50:07 +0000
957@@ -0,0 +1,32 @@
958+#!/bin/bash
959+#--------------------------------------------
960+# This file is managed by Juju
961+#--------------------------------------------
962+#
963+# Copyright 2009,2012 Canonical Ltd.
964+# Author: Tom Haddon
965+
966+CRITICAL=0
967+NOTACTIVE=''
968+LOGFILE=/var/log/nagios/check_haproxy.log
969+AUTH=$(grep -r "stats auth" /etc/haproxy | head -1 | awk '{print $4}')
970+
971+for appserver in $(grep ' server' /etc/haproxy/haproxy.cfg | awk '{print $2'});
972+do
973+ output=$(/usr/lib/nagios/plugins/check_http -a ${AUTH} -I 127.0.0.1 -p 8888 --regex="class=\"(active|backup)(2|3).*${appserver}" -e ' 200 OK')
974+ if [ $? != 0 ]; then
975+ date >> $LOGFILE
976+ echo $output >> $LOGFILE
977+ /usr/lib/nagios/plugins/check_http -a ${AUTH} -I 127.0.0.1 -p 8888 -v | grep $appserver >> $LOGFILE 2>&1
978+ CRITICAL=1
979+ NOTACTIVE="${NOTACTIVE} $appserver"
980+ fi
981+done
982+
983+if [ $CRITICAL = 1 ]; then
984+ echo "CRITICAL:${NOTACTIVE}"
985+ exit 2
986+fi
987+
988+echo "OK: All haproxy instances looking good"
989+exit 0
990
991=== added file 'hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh'
992--- hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh 1970-01-01 00:00:00 +0000
993+++ hooks/charmhelpers/contrib/openstack/files/check_haproxy_queue_depth.sh 2015-04-16 21:50:07 +0000
994@@ -0,0 +1,30 @@
995+#!/bin/bash
996+#--------------------------------------------
997+# This file is managed by Juju
998+#--------------------------------------------
999+#
1000+# Copyright 2009,2012 Canonical Ltd.
1001+# Author: Tom Haddon
1002+
1003+# These should be config options at some stage
1004+CURRQthrsh=0
1005+MAXQthrsh=100
1006+
1007+AUTH=$(grep -r "stats auth" /etc/haproxy | head -1 | awk '{print $4}')
1008+
1009+HAPROXYSTATS=$(/usr/lib/nagios/plugins/check_http -a ${AUTH} -I 127.0.0.1 -p 8888 -u '/;csv' -v)
1010+
1011+for BACKEND in $(echo $HAPROXYSTATS| xargs -n1 | grep BACKEND | awk -F , '{print $1}')
1012+do
1013+ CURRQ=$(echo "$HAPROXYSTATS" | grep $BACKEND | grep BACKEND | cut -d , -f 3)
1014+ MAXQ=$(echo "$HAPROXYSTATS" | grep $BACKEND | grep BACKEND | cut -d , -f 4)
1015+
1016+ if [[ $CURRQ -gt $CURRQthrsh || $MAXQ -gt $MAXQthrsh ]] ; then
1017+ echo "CRITICAL: queue depth for $BACKEND - CURRENT:$CURRQ MAX:$MAXQ"
1018+ exit 2
1019+ fi
1020+done
1021+
1022+echo "OK: All haproxy queue depths looking good"
1023+exit 0
1024+
1025
1026=== modified file 'hooks/charmhelpers/contrib/openstack/ip.py'
1027--- hooks/charmhelpers/contrib/openstack/ip.py 2015-01-26 09:45:23 +0000
1028+++ hooks/charmhelpers/contrib/openstack/ip.py 2015-04-16 21:50:07 +0000
1029@@ -26,6 +26,8 @@
1030 )
1031 from charmhelpers.contrib.hahelpers.cluster import is_clustered
1032
1033+from functools import partial
1034+
1035 PUBLIC = 'public'
1036 INTERNAL = 'int'
1037 ADMIN = 'admin'
1038@@ -107,3 +109,38 @@
1039 "clustered=%s)" % (net_type, clustered))
1040
1041 return resolved_address
1042+
1043+
1044+def endpoint_url(configs, url_template, port, endpoint_type=PUBLIC,
1045+ override=None):
1046+ """Returns the correct endpoint URL to advertise to Keystone.
1047+
1048+ This method provides the correct endpoint URL which should be advertised to
1049+ the keystone charm for endpoint creation. This method allows for the url to
1050+ be overridden to force a keystone endpoint to have specific URL for any of
1051+ the defined scopes (admin, internal, public).
1052+
1053+ :param configs: OSTemplateRenderer config templating object to inspect
1054+ for a complete https context.
1055+ :param url_template: str format string for creating the url template. Only
1056+ two values will be passed - the scheme+hostname
1057+ returned by the canonical_url and the port.
1058+ :param endpoint_type: str endpoint type to resolve.
1059+ :param override: str the name of the config option which overrides the
1060+ endpoint URL defined by the charm itself. None will
1061+ disable any overrides (default).
1062+ """
1063+ if override:
1064+ # Return any user-defined overrides for the keystone endpoint URL.
1065+ user_value = config(override)
1066+ if user_value:
1067+ return user_value.strip()
1068+
1069+ return url_template % (canonical_url(configs, endpoint_type), port)
1070+
1071+
1072+public_endpoint = partial(endpoint_url, endpoint_type=PUBLIC)
1073+
1074+internal_endpoint = partial(endpoint_url, endpoint_type=INTERNAL)
1075+
1076+admin_endpoint = partial(endpoint_url, endpoint_type=ADMIN)
1077
1078=== modified file 'hooks/charmhelpers/contrib/openstack/neutron.py'
1079--- hooks/charmhelpers/contrib/openstack/neutron.py 2015-01-26 09:45:23 +0000
1080+++ hooks/charmhelpers/contrib/openstack/neutron.py 2015-04-16 21:50:07 +0000
1081@@ -16,6 +16,7 @@
1082
1083 # Various utilies for dealing with Neutron and the renaming from Quantum.
1084
1085+import six
1086 from subprocess import check_output
1087
1088 from charmhelpers.core.hookenv import (
1089@@ -179,6 +180,19 @@
1090 'nova-api-metadata']],
1091 'server_packages': ['neutron-server', 'calico-control'],
1092 'server_services': ['neutron-server']
1093+ },
1094+ 'vsp': {
1095+ 'config': '/etc/neutron/plugins/nuage/nuage_plugin.ini',
1096+ 'driver': 'neutron.plugins.nuage.plugin.NuagePlugin',
1097+ 'contexts': [
1098+ context.SharedDBContext(user=config('neutron-database-user'),
1099+ database=config('neutron-database'),
1100+ relation_prefix='neutron',
1101+ ssl_dir=NEUTRON_CONF_DIR)],
1102+ 'services': [],
1103+ 'packages': [],
1104+ 'server_packages': ['neutron-server', 'neutron-plugin-nuage'],
1105+ 'server_services': ['neutron-server']
1106 }
1107 }
1108 if release >= 'icehouse':
1109@@ -237,3 +251,72 @@
1110 else:
1111 # ensure accurate naming for all releases post-H
1112 return 'neutron'
1113+
1114+
1115+def parse_mappings(mappings):
1116+ parsed = {}
1117+ if mappings:
1118+ mappings = mappings.split(' ')
1119+ for m in mappings:
1120+ p = m.partition(':')
1121+ if p[1] == ':':
1122+ parsed[p[0].strip()] = p[2].strip()
1123+
1124+ return parsed
1125+
1126+
1127+def parse_bridge_mappings(mappings):
1128+ """Parse bridge mappings.
1129+
1130+ Mappings must be a space-delimited list of provider:bridge mappings.
1131+
1132+ Returns dict of the form {provider:bridge}.
1133+ """
1134+ return parse_mappings(mappings)
1135+
1136+
1137+def parse_data_port_mappings(mappings, default_bridge='br-data'):
1138+ """Parse data port mappings.
1139+
1140+ Mappings must be a space-delimited list of bridge:port mappings.
1141+
1142+ Returns dict of the form {bridge:port}.
1143+ """
1144+ _mappings = parse_mappings(mappings)
1145+ if not _mappings:
1146+ if not mappings:
1147+ return {}
1148+
1149+ # For backwards-compatibility we need to support port-only provided in
1150+ # config.
1151+ _mappings = {default_bridge: mappings.split(' ')[0]}
1152+
1153+ bridges = _mappings.keys()
1154+ ports = _mappings.values()
1155+ if len(set(bridges)) != len(bridges):
1156+ raise Exception("It is not allowed to have more than one port "
1157+ "configured on the same bridge")
1158+
1159+ if len(set(ports)) != len(ports):
1160+ raise Exception("It is not allowed to have the same port configured "
1161+ "on more than one bridge")
1162+
1163+ return _mappings
1164+
1165+
1166+def parse_vlan_range_mappings(mappings):
1167+ """Parse vlan range mappings.
1168+
1169+ Mappings must be a space-delimited list of provider:start:end mappings.
1170+
1171+ Returns dict of the form {provider: (start, end)}.
1172+ """
1173+ _mappings = parse_mappings(mappings)
1174+ if not _mappings:
1175+ return {}
1176+
1177+ mappings = {}
1178+ for p, r in six.iteritems(_mappings):
1179+ mappings[p] = tuple(r.split(':'))
1180+
1181+ return mappings
1182
1183=== added file 'hooks/charmhelpers/contrib/openstack/templates/git.upstart'
1184--- hooks/charmhelpers/contrib/openstack/templates/git.upstart 1970-01-01 00:00:00 +0000
1185+++ hooks/charmhelpers/contrib/openstack/templates/git.upstart 2015-04-16 21:50:07 +0000
1186@@ -0,0 +1,17 @@
1187+description "{{ service_description }}"
1188+author "Juju {{ service_name }} Charm <juju@localhost>"
1189+
1190+start on runlevel [2345]
1191+stop on runlevel [!2345]
1192+
1193+respawn
1194+
1195+exec start-stop-daemon --start --chuid {{ user_name }} \
1196+ --chdir {{ start_dir }} --name {{ process_name }} \
1197+ --exec {{ executable_name }} -- \
1198+ {% for config_file in config_files -%}
1199+ --config-file={{ config_file }} \
1200+ {% endfor -%}
1201+ {% if log_file -%}
1202+ --log-file={{ log_file }}
1203+ {% endif -%}
1204
1205=== added file 'hooks/charmhelpers/contrib/openstack/templates/section-keystone-authtoken'
1206--- hooks/charmhelpers/contrib/openstack/templates/section-keystone-authtoken 1970-01-01 00:00:00 +0000
1207+++ hooks/charmhelpers/contrib/openstack/templates/section-keystone-authtoken 2015-04-16 21:50:07 +0000
1208@@ -0,0 +1,9 @@
1209+{% if auth_host -%}
1210+[keystone_authtoken]
1211+identity_uri = {{ auth_protocol }}://{{ auth_host }}:{{ auth_port }}/{{ auth_admin_prefix }}
1212+auth_uri = {{ service_protocol }}://{{ service_host }}:{{ service_port }}/{{ service_admin_prefix }}
1213+admin_tenant_name = {{ admin_tenant_name }}
1214+admin_user = {{ admin_user }}
1215+admin_password = {{ admin_password }}
1216+signing_dir = {{ signing_dir }}
1217+{% endif -%}
1218
1219=== added file 'hooks/charmhelpers/contrib/openstack/templates/section-rabbitmq-oslo'
1220--- hooks/charmhelpers/contrib/openstack/templates/section-rabbitmq-oslo 1970-01-01 00:00:00 +0000
1221+++ hooks/charmhelpers/contrib/openstack/templates/section-rabbitmq-oslo 2015-04-16 21:50:07 +0000
1222@@ -0,0 +1,22 @@
1223+{% if rabbitmq_host or rabbitmq_hosts -%}
1224+[oslo_messaging_rabbit]
1225+rabbit_userid = {{ rabbitmq_user }}
1226+rabbit_virtual_host = {{ rabbitmq_virtual_host }}
1227+rabbit_password = {{ rabbitmq_password }}
1228+{% if rabbitmq_hosts -%}
1229+rabbit_hosts = {{ rabbitmq_hosts }}
1230+{% if rabbitmq_ha_queues -%}
1231+rabbit_ha_queues = True
1232+rabbit_durable_queues = False
1233+{% endif -%}
1234+{% else -%}
1235+rabbit_host = {{ rabbitmq_host }}
1236+{% endif -%}
1237+{% if rabbit_ssl_port -%}
1238+rabbit_use_ssl = True
1239+rabbit_port = {{ rabbit_ssl_port }}
1240+{% if rabbit_ssl_ca -%}
1241+kombu_ssl_ca_certs = {{ rabbit_ssl_ca }}
1242+{% endif -%}
1243+{% endif -%}
1244+{% endif -%}
1245
1246=== added file 'hooks/charmhelpers/contrib/openstack/templates/section-zeromq'
1247--- hooks/charmhelpers/contrib/openstack/templates/section-zeromq 1970-01-01 00:00:00 +0000
1248+++ hooks/charmhelpers/contrib/openstack/templates/section-zeromq 2015-04-16 21:50:07 +0000
1249@@ -0,0 +1,14 @@
1250+{% if zmq_host -%}
1251+# ZeroMQ configuration (restart-nonce: {{ zmq_nonce }})
1252+rpc_backend = zmq
1253+rpc_zmq_host = {{ zmq_host }}
1254+{% if zmq_redis_address -%}
1255+rpc_zmq_matchmaker = redis
1256+matchmaker_heartbeat_freq = 15
1257+matchmaker_heartbeat_ttl = 30
1258+[matchmaker_redis]
1259+host = {{ zmq_redis_address }}
1260+{% else -%}
1261+rpc_zmq_matchmaker = ring
1262+{% endif -%}
1263+{% endif -%}
1264
1265=== modified file 'hooks/charmhelpers/contrib/openstack/utils.py'
1266--- hooks/charmhelpers/contrib/openstack/utils.py 2015-01-26 09:45:23 +0000
1267+++ hooks/charmhelpers/contrib/openstack/utils.py 2015-04-16 21:50:07 +0000
1268@@ -23,12 +23,17 @@
1269 import subprocess
1270 import json
1271 import os
1272-import socket
1273 import sys
1274
1275 import six
1276 import yaml
1277
1278+from charmhelpers.contrib.network import ip
1279+
1280+from charmhelpers.core import (
1281+ unitdata,
1282+)
1283+
1284 from charmhelpers.core.hookenv import (
1285 config,
1286 log as juju_log,
1287@@ -103,6 +108,7 @@
1288 ('2.1.0', 'juno'),
1289 ('2.2.0', 'juno'),
1290 ('2.2.1', 'kilo'),
1291+ ('2.2.2', 'kilo'),
1292 ])
1293
1294 DEFAULT_LOOPBACK_SIZE = '5G'
1295@@ -328,6 +334,21 @@
1296 error_out("Invalid openstack-release specified: %s" % rel)
1297
1298
1299+def config_value_changed(option):
1300+ """
1301+ Determine if config value changed since last call to this function.
1302+ """
1303+ hook_data = unitdata.HookData()
1304+ with hook_data():
1305+ db = unitdata.kv()
1306+ current = config(option)
1307+ saved = db.get(option)
1308+ db.set(option, current)
1309+ if saved is None:
1310+ return False
1311+ return current != saved
1312+
1313+
1314 def save_script_rc(script_path="scripts/scriptrc", **env_vars):
1315 """
1316 Write an rc file in the charm-delivered directory containing
1317@@ -420,77 +441,10 @@
1318 else:
1319 zap_disk(block_device)
1320
1321-
1322-def is_ip(address):
1323- """
1324- Returns True if address is a valid IP address.
1325- """
1326- try:
1327- # Test to see if already an IPv4 address
1328- socket.inet_aton(address)
1329- return True
1330- except socket.error:
1331- return False
1332-
1333-
1334-def ns_query(address):
1335- try:
1336- import dns.resolver
1337- except ImportError:
1338- apt_install('python-dnspython')
1339- import dns.resolver
1340-
1341- if isinstance(address, dns.name.Name):
1342- rtype = 'PTR'
1343- elif isinstance(address, six.string_types):
1344- rtype = 'A'
1345- else:
1346- return None
1347-
1348- answers = dns.resolver.query(address, rtype)
1349- if answers:
1350- return str(answers[0])
1351- return None
1352-
1353-
1354-def get_host_ip(hostname):
1355- """
1356- Resolves the IP for a given hostname, or returns
1357- the input if it is already an IP.
1358- """
1359- if is_ip(hostname):
1360- return hostname
1361-
1362- return ns_query(hostname)
1363-
1364-
1365-def get_hostname(address, fqdn=True):
1366- """
1367- Resolves hostname for given IP, or returns the input
1368- if it is already a hostname.
1369- """
1370- if is_ip(address):
1371- try:
1372- import dns.reversename
1373- except ImportError:
1374- apt_install('python-dnspython')
1375- import dns.reversename
1376-
1377- rev = dns.reversename.from_address(address)
1378- result = ns_query(rev)
1379- if not result:
1380- return None
1381- else:
1382- result = address
1383-
1384- if fqdn:
1385- # strip trailing .
1386- if result.endswith('.'):
1387- return result[:-1]
1388- else:
1389- return result
1390- else:
1391- return result.split('.')[0]
1392+is_ip = ip.is_ip
1393+ns_query = ip.ns_query
1394+get_host_ip = ip.get_host_ip
1395+get_hostname = ip.get_hostname
1396
1397
1398 def get_matchmaker_map(mm_file='/etc/oslo/matchmaker_ring.json'):
1399@@ -534,82 +488,106 @@
1400
1401
1402 def git_install_requested():
1403- """Returns true if openstack-origin-git is specified."""
1404- return config('openstack-origin-git') != "None"
1405+ """
1406+ Returns true if openstack-origin-git is specified.
1407+ """
1408+ return config('openstack-origin-git') is not None
1409
1410
1411 requirements_dir = None
1412
1413
1414-def git_clone_and_install(file_name, core_project):
1415- """Clone/install all OpenStack repos specified in yaml config file."""
1416+def git_clone_and_install(projects_yaml, core_project):
1417+ """
1418+ Clone/install all specified OpenStack repositories.
1419+
1420+ The expected format of projects_yaml is:
1421+ repositories:
1422+ - {name: keystone,
1423+ repository: 'git://git.openstack.org/openstack/keystone.git',
1424+ branch: 'stable/icehouse'}
1425+ - {name: requirements,
1426+ repository: 'git://git.openstack.org/openstack/requirements.git',
1427+ branch: 'stable/icehouse'}
1428+ directory: /mnt/openstack-git
1429+ http_proxy: http://squid.internal:3128
1430+ https_proxy: https://squid.internal:3128
1431+
1432+ The directory, http_proxy, and https_proxy keys are optional.
1433+ """
1434 global requirements_dir
1435+ parent_dir = '/mnt/openstack-git'
1436
1437- if file_name == "None":
1438+ if not projects_yaml:
1439 return
1440
1441- yaml_file = os.path.join(charm_dir(), file_name)
1442-
1443- # clone/install the requirements project first
1444- installed = _git_clone_and_install_subset(yaml_file,
1445- whitelist=['requirements'])
1446- if 'requirements' not in installed:
1447- error_out('requirements git repository must be specified')
1448-
1449- # clone/install all other projects except requirements and the core project
1450- blacklist = ['requirements', core_project]
1451- _git_clone_and_install_subset(yaml_file, blacklist=blacklist,
1452- update_requirements=True)
1453-
1454- # clone/install the core project
1455- whitelist = [core_project]
1456- installed = _git_clone_and_install_subset(yaml_file, whitelist=whitelist,
1457- update_requirements=True)
1458- if core_project not in installed:
1459- error_out('{} git repository must be specified'.format(core_project))
1460-
1461-
1462-def _git_clone_and_install_subset(yaml_file, whitelist=[], blacklist=[],
1463- update_requirements=False):
1464- """Clone/install subset of OpenStack repos specified in yaml config file."""
1465- global requirements_dir
1466- installed = []
1467-
1468- with open(yaml_file, 'r') as fd:
1469- projects = yaml.load(fd)
1470- for proj, val in projects.items():
1471- # The project subset is chosen based on the following 3 rules:
1472- # 1) If project is in blacklist, we don't clone/install it, period.
1473- # 2) If whitelist is empty, we clone/install everything else.
1474- # 3) If whitelist is not empty, we clone/install everything in the
1475- # whitelist.
1476- if proj in blacklist:
1477- continue
1478- if whitelist and proj not in whitelist:
1479- continue
1480- repo = val['repository']
1481- branch = val['branch']
1482- repo_dir = _git_clone_and_install_single(repo, branch,
1483- update_requirements)
1484- if proj == 'requirements':
1485- requirements_dir = repo_dir
1486- installed.append(proj)
1487- return installed
1488-
1489-
1490-def _git_clone_and_install_single(repo, branch, update_requirements=False):
1491- """Clone and install a single git repository."""
1492- dest_parent_dir = "/mnt/openstack-git/"
1493- dest_dir = os.path.join(dest_parent_dir, os.path.basename(repo))
1494-
1495- if not os.path.exists(dest_parent_dir):
1496- juju_log('Host dir not mounted at {}. '
1497- 'Creating directory there instead.'.format(dest_parent_dir))
1498- os.mkdir(dest_parent_dir)
1499+ projects = yaml.load(projects_yaml)
1500+ _git_validate_projects_yaml(projects, core_project)
1501+
1502+ old_environ = dict(os.environ)
1503+
1504+ if 'http_proxy' in projects.keys():
1505+ os.environ['http_proxy'] = projects['http_proxy']
1506+ if 'https_proxy' in projects.keys():
1507+ os.environ['https_proxy'] = projects['https_proxy']
1508+
1509+ if 'directory' in projects.keys():
1510+ parent_dir = projects['directory']
1511+
1512+ for p in projects['repositories']:
1513+ repo = p['repository']
1514+ branch = p['branch']
1515+ if p['name'] == 'requirements':
1516+ repo_dir = _git_clone_and_install_single(repo, branch, parent_dir,
1517+ update_requirements=False)
1518+ requirements_dir = repo_dir
1519+ else:
1520+ repo_dir = _git_clone_and_install_single(repo, branch, parent_dir,
1521+ update_requirements=True)
1522+
1523+ os.environ = old_environ
1524+
1525+
1526+def _git_validate_projects_yaml(projects, core_project):
1527+ """
1528+ Validate the projects yaml.
1529+ """
1530+ _git_ensure_key_exists('repositories', projects)
1531+
1532+ for project in projects['repositories']:
1533+ _git_ensure_key_exists('name', project.keys())
1534+ _git_ensure_key_exists('repository', project.keys())
1535+ _git_ensure_key_exists('branch', project.keys())
1536+
1537+ if projects['repositories'][0]['name'] != 'requirements':
1538+ error_out('{} git repo must be specified first'.format('requirements'))
1539+
1540+ if projects['repositories'][-1]['name'] != core_project:
1541+ error_out('{} git repo must be specified last'.format(core_project))
1542+
1543+
1544+def _git_ensure_key_exists(key, keys):
1545+ """
1546+ Ensure that key exists in keys.
1547+ """
1548+ if key not in keys:
1549+ error_out('openstack-origin-git key \'{}\' is missing'.format(key))
1550+
1551+
1552+def _git_clone_and_install_single(repo, branch, parent_dir, update_requirements):
1553+ """
1554+ Clone and install a single git repository.
1555+ """
1556+ dest_dir = os.path.join(parent_dir, os.path.basename(repo))
1557+
1558+ if not os.path.exists(parent_dir):
1559+ juju_log('Directory already exists at {}. '
1560+ 'No need to create directory.'.format(parent_dir))
1561+ os.mkdir(parent_dir)
1562
1563 if not os.path.exists(dest_dir):
1564 juju_log('Cloning git repo: {}, branch: {}'.format(repo, branch))
1565- repo_dir = install_remote(repo, dest=dest_parent_dir, branch=branch)
1566+ repo_dir = install_remote(repo, dest=parent_dir, branch=branch)
1567 else:
1568 repo_dir = dest_dir
1569
1570@@ -626,16 +604,39 @@
1571
1572
1573 def _git_update_requirements(package_dir, reqs_dir):
1574- """Update from global requirements.
1575+ """
1576+ Update from global requirements.
1577
1578- Update an OpenStack git directory's requirements.txt and
1579- test-requirements.txt from global-requirements.txt."""
1580+ Update an OpenStack git directory's requirements.txt and
1581+ test-requirements.txt from global-requirements.txt.
1582+ """
1583 orig_dir = os.getcwd()
1584 os.chdir(reqs_dir)
1585- cmd = "python update.py {}".format(package_dir)
1586+ cmd = ['python', 'update.py', package_dir]
1587 try:
1588- subprocess.check_call(cmd.split(' '))
1589+ subprocess.check_call(cmd)
1590 except subprocess.CalledProcessError:
1591 package = os.path.basename(package_dir)
1592 error_out("Error updating {} from global-requirements.txt".format(package))
1593 os.chdir(orig_dir)
1594+
1595+
1596+def git_src_dir(projects_yaml, project):
1597+ """
1598+ Return the directory where the specified project's source is located.
1599+ """
1600+ parent_dir = '/mnt/openstack-git'
1601+
1602+ if not projects_yaml:
1603+ return
1604+
1605+ projects = yaml.load(projects_yaml)
1606+
1607+ if 'directory' in projects.keys():
1608+ parent_dir = projects['directory']
1609+
1610+ for p in projects['repositories']:
1611+ if p['name'] == project:
1612+ return os.path.join(parent_dir, os.path.basename(p['repository']))
1613+
1614+ return None
1615
1616=== modified file 'hooks/charmhelpers/contrib/python/packages.py'
1617--- hooks/charmhelpers/contrib/python/packages.py 2015-01-26 09:45:23 +0000
1618+++ hooks/charmhelpers/contrib/python/packages.py 2015-04-16 21:50:07 +0000
1619@@ -17,8 +17,6 @@
1620 # You should have received a copy of the GNU Lesser General Public License
1621 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1622
1623-__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
1624-
1625 from charmhelpers.fetch import apt_install, apt_update
1626 from charmhelpers.core.hookenv import log
1627
1628@@ -29,6 +27,8 @@
1629 apt_install('python-pip')
1630 from pip import main as pip_execute
1631
1632+__author__ = "Jorge Niedbalski <jorge.niedbalski@canonical.com>"
1633+
1634
1635 def parse_options(given, available):
1636 """Given a set of options, check if available"""
1637
1638=== modified file 'hooks/charmhelpers/core/fstab.py'
1639--- hooks/charmhelpers/core/fstab.py 2015-01-26 09:45:23 +0000
1640+++ hooks/charmhelpers/core/fstab.py 2015-04-16 21:50:07 +0000
1641@@ -17,11 +17,11 @@
1642 # You should have received a copy of the GNU Lesser General Public License
1643 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1644
1645-__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1646-
1647 import io
1648 import os
1649
1650+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1651+
1652
1653 class Fstab(io.FileIO):
1654 """This class extends file in order to implement a file reader/writer
1655@@ -77,7 +77,7 @@
1656 for line in self.readlines():
1657 line = line.decode('us-ascii')
1658 try:
1659- if line.strip() and not line.startswith("#"):
1660+ if line.strip() and not line.strip().startswith("#"):
1661 yield self._hydrate_entry(line)
1662 except ValueError:
1663 pass
1664@@ -104,7 +104,7 @@
1665
1666 found = False
1667 for index, line in enumerate(lines):
1668- if not line.startswith("#"):
1669+ if line.strip() and not line.strip().startswith("#"):
1670 if self._hydrate_entry(line) == entry:
1671 found = True
1672 break
1673
1674=== modified file 'hooks/charmhelpers/core/hookenv.py'
1675--- hooks/charmhelpers/core/hookenv.py 2015-01-26 09:45:23 +0000
1676+++ hooks/charmhelpers/core/hookenv.py 2015-04-16 21:50:07 +0000
1677@@ -20,11 +20,13 @@
1678 # Authors:
1679 # Charm Helpers Developers <juju@lists.ubuntu.com>
1680
1681+from __future__ import print_function
1682 import os
1683 import json
1684 import yaml
1685 import subprocess
1686 import sys
1687+import errno
1688 from subprocess import CalledProcessError
1689
1690 import six
1691@@ -87,7 +89,18 @@
1692 if not isinstance(message, six.string_types):
1693 message = repr(message)
1694 command += [message]
1695- subprocess.call(command)
1696+ # Missing juju-log should not cause failures in unit tests
1697+ # Send log output to stderr
1698+ try:
1699+ subprocess.call(command)
1700+ except OSError as e:
1701+ if e.errno == errno.ENOENT:
1702+ if level:
1703+ message = "{}: {}".format(level, message)
1704+ message = "juju-log: {}".format(message)
1705+ print(message, file=sys.stderr)
1706+ else:
1707+ raise
1708
1709
1710 class Serializable(UserDict):
1711@@ -566,3 +579,29 @@
1712 def charm_dir():
1713 """Return the root directory of the current charm"""
1714 return os.environ.get('CHARM_DIR')
1715+
1716+
1717+@cached
1718+def action_get(key=None):
1719+ """Gets the value of an action parameter, or all key/value param pairs"""
1720+ cmd = ['action-get']
1721+ if key is not None:
1722+ cmd.append(key)
1723+ cmd.append('--format=json')
1724+ action_data = json.loads(subprocess.check_output(cmd).decode('UTF-8'))
1725+ return action_data
1726+
1727+
1728+def action_set(values):
1729+ """Sets the values to be returned after the action finishes"""
1730+ cmd = ['action-set']
1731+ for k, v in list(values.items()):
1732+ cmd.append('{}={}'.format(k, v))
1733+ subprocess.check_call(cmd)
1734+
1735+
1736+def action_fail(message):
1737+ """Sets the action status to failed and sets the error message.
1738+
1739+ The results set by action_set are preserved."""
1740+ subprocess.check_call(['action-fail', message])
1741
1742=== modified file 'hooks/charmhelpers/core/host.py'
1743--- hooks/charmhelpers/core/host.py 2015-01-26 09:45:23 +0000
1744+++ hooks/charmhelpers/core/host.py 2015-04-16 21:50:07 +0000
1745@@ -191,11 +191,11 @@
1746
1747
1748 def write_file(path, content, owner='root', group='root', perms=0o444):
1749- """Create or overwrite a file with the contents of a string"""
1750+ """Create or overwrite a file with the contents of a byte string."""
1751 log("Writing file {} {}:{} {:o}".format(path, owner, group, perms))
1752 uid = pwd.getpwnam(owner).pw_uid
1753 gid = grp.getgrnam(group).gr_gid
1754- with open(path, 'w') as target:
1755+ with open(path, 'wb') as target:
1756 os.fchown(target.fileno(), uid, gid)
1757 os.fchmod(target.fileno(), perms)
1758 target.write(content)
1759@@ -305,11 +305,11 @@
1760 ceph_client_changed function.
1761 """
1762 def wrap(f):
1763- def wrapped_f(*args):
1764+ def wrapped_f(*args, **kwargs):
1765 checksums = {}
1766 for path in restart_map:
1767 checksums[path] = file_hash(path)
1768- f(*args)
1769+ f(*args, **kwargs)
1770 restarts = []
1771 for path in restart_map:
1772 if checksums[path] != file_hash(path):
1773@@ -339,12 +339,16 @@
1774 def pwgen(length=None):
1775 """Generate a random pasword."""
1776 if length is None:
1777+ # A random length is ok to use a weak PRNG
1778 length = random.choice(range(35, 45))
1779 alphanumeric_chars = [
1780 l for l in (string.ascii_letters + string.digits)
1781 if l not in 'l0QD1vAEIOUaeiou']
1782+ # Use a crypto-friendly PRNG (e.g. /dev/urandom) for making the
1783+ # actual password
1784+ random_generator = random.SystemRandom()
1785 random_chars = [
1786- random.choice(alphanumeric_chars) for _ in range(length)]
1787+ random_generator.choice(alphanumeric_chars) for _ in range(length)]
1788 return(''.join(random_chars))
1789
1790
1791@@ -361,7 +365,7 @@
1792 ip_output = (line for line in ip_output if line)
1793 for line in ip_output:
1794 if line.split()[1].startswith(int_type):
1795- matched = re.search('.*: (bond[0-9]+\.[0-9]+)@.*', line)
1796+ matched = re.search('.*: (' + int_type + r'[0-9]+\.[0-9]+)@.*', line)
1797 if matched:
1798 interface = matched.groups()[0]
1799 else:
1800
1801=== modified file 'hooks/charmhelpers/core/services/helpers.py'
1802--- hooks/charmhelpers/core/services/helpers.py 2015-01-26 09:45:23 +0000
1803+++ hooks/charmhelpers/core/services/helpers.py 2015-04-16 21:50:07 +0000
1804@@ -45,12 +45,14 @@
1805 """
1806 name = None
1807 interface = None
1808- required_keys = []
1809
1810 def __init__(self, name=None, additional_required_keys=None):
1811+ if not hasattr(self, 'required_keys'):
1812+ self.required_keys = []
1813+
1814 if name is not None:
1815 self.name = name
1816- if additional_required_keys is not None:
1817+ if additional_required_keys:
1818 self.required_keys.extend(additional_required_keys)
1819 self.get_data()
1820
1821@@ -134,7 +136,10 @@
1822 """
1823 name = 'db'
1824 interface = 'mysql'
1825- required_keys = ['host', 'user', 'password', 'database']
1826+
1827+ def __init__(self, *args, **kwargs):
1828+ self.required_keys = ['host', 'user', 'password', 'database']
1829+ RelationContext.__init__(self, *args, **kwargs)
1830
1831
1832 class HttpRelation(RelationContext):
1833@@ -146,7 +151,10 @@
1834 """
1835 name = 'website'
1836 interface = 'http'
1837- required_keys = ['host', 'port']
1838+
1839+ def __init__(self, *args, **kwargs):
1840+ self.required_keys = ['host', 'port']
1841+ RelationContext.__init__(self, *args, **kwargs)
1842
1843 def provide_data(self):
1844 return {
1845
1846=== added file 'hooks/charmhelpers/core/strutils.py'
1847--- hooks/charmhelpers/core/strutils.py 1970-01-01 00:00:00 +0000
1848+++ hooks/charmhelpers/core/strutils.py 2015-04-16 21:50:07 +0000
1849@@ -0,0 +1,42 @@
1850+#!/usr/bin/env python
1851+# -*- coding: utf-8 -*-
1852+
1853+# Copyright 2014-2015 Canonical Limited.
1854+#
1855+# This file is part of charm-helpers.
1856+#
1857+# charm-helpers is free software: you can redistribute it and/or modify
1858+# it under the terms of the GNU Lesser General Public License version 3 as
1859+# published by the Free Software Foundation.
1860+#
1861+# charm-helpers is distributed in the hope that it will be useful,
1862+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1863+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1864+# GNU Lesser General Public License for more details.
1865+#
1866+# You should have received a copy of the GNU Lesser General Public License
1867+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1868+
1869+import six
1870+
1871+
1872+def bool_from_string(value):
1873+ """Interpret string value as boolean.
1874+
1875+ Returns True if value translates to True otherwise False.
1876+ """
1877+ if isinstance(value, six.string_types):
1878+ value = six.text_type(value)
1879+ else:
1880+ msg = "Unable to interpret non-string value '%s' as boolean" % (value)
1881+ raise ValueError(msg)
1882+
1883+ value = value.strip().lower()
1884+
1885+ if value in ['y', 'yes', 'true', 't', 'on']:
1886+ return True
1887+ elif value in ['n', 'no', 'false', 'f', 'off']:
1888+ return False
1889+
1890+ msg = "Unable to interpret string value '%s' as boolean" % (value)
1891+ raise ValueError(msg)
1892
1893=== modified file 'hooks/charmhelpers/core/sysctl.py'
1894--- hooks/charmhelpers/core/sysctl.py 2015-03-05 10:50:47 +0000
1895+++ hooks/charmhelpers/core/sysctl.py 2015-04-16 21:50:07 +0000
1896@@ -17,8 +17,6 @@
1897 # You should have received a copy of the GNU Lesser General Public License
1898 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1899
1900-__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1901-
1902 import yaml
1903
1904 from subprocess import check_call
1905@@ -29,6 +27,8 @@
1906 ERROR,
1907 )
1908
1909+__author__ = 'Jorge Niedbalski R. <jorge.niedbalski@canonical.com>'
1910+
1911
1912 def create(sysctl_dict, sysctl_file):
1913 """Creates a sysctl.conf file from a YAML associative array
1914
1915=== modified file 'hooks/charmhelpers/core/templating.py'
1916--- hooks/charmhelpers/core/templating.py 2015-01-26 09:45:23 +0000
1917+++ hooks/charmhelpers/core/templating.py 2015-04-16 21:50:07 +0000
1918@@ -21,7 +21,7 @@
1919
1920
1921 def render(source, target, context, owner='root', group='root',
1922- perms=0o444, templates_dir=None):
1923+ perms=0o444, templates_dir=None, encoding='UTF-8'):
1924 """
1925 Render a template.
1926
1927@@ -64,5 +64,5 @@
1928 level=hookenv.ERROR)
1929 raise e
1930 content = template.render(context)
1931- host.mkdir(os.path.dirname(target), owner, group)
1932- host.write_file(target, content, owner, group, perms)
1933+ host.mkdir(os.path.dirname(target), owner, group, perms=0o755)
1934+ host.write_file(target, content.encode(encoding), owner, group, perms)
1935
1936=== added file 'hooks/charmhelpers/core/unitdata.py'
1937--- hooks/charmhelpers/core/unitdata.py 1970-01-01 00:00:00 +0000
1938+++ hooks/charmhelpers/core/unitdata.py 2015-04-16 21:50:07 +0000
1939@@ -0,0 +1,477 @@
1940+#!/usr/bin/env python
1941+# -*- coding: utf-8 -*-
1942+#
1943+# Copyright 2014-2015 Canonical Limited.
1944+#
1945+# This file is part of charm-helpers.
1946+#
1947+# charm-helpers is free software: you can redistribute it and/or modify
1948+# it under the terms of the GNU Lesser General Public License version 3 as
1949+# published by the Free Software Foundation.
1950+#
1951+# charm-helpers is distributed in the hope that it will be useful,
1952+# but WITHOUT ANY WARRANTY; without even the implied warranty of
1953+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1954+# GNU Lesser General Public License for more details.
1955+#
1956+# You should have received a copy of the GNU Lesser General Public License
1957+# along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
1958+#
1959+#
1960+# Authors:
1961+# Kapil Thangavelu <kapil.foss@gmail.com>
1962+#
1963+"""
1964+Intro
1965+-----
1966+
1967+A simple way to store state in units. This provides a key value
1968+storage with support for versioned, transactional operation,
1969+and can calculate deltas from previous values to simplify unit logic
1970+when processing changes.
1971+
1972+
1973+Hook Integration
1974+----------------
1975+
1976+There are several extant frameworks for hook execution, including
1977+
1978+ - charmhelpers.core.hookenv.Hooks
1979+ - charmhelpers.core.services.ServiceManager
1980+
1981+The storage classes are framework agnostic, one simple integration is
1982+via the HookData contextmanager. It will record the current hook
1983+execution environment (including relation data, config data, etc.),
1984+setup a transaction and allow easy access to the changes from
1985+previously seen values. One consequence of the integration is the
1986+reservation of particular keys ('rels', 'unit', 'env', 'config',
1987+'charm_revisions') for their respective values.
1988+
1989+Here's a fully worked integration example using hookenv.Hooks::
1990+
1991+ from charmhelper.core import hookenv, unitdata
1992+
1993+ hook_data = unitdata.HookData()
1994+ db = unitdata.kv()
1995+ hooks = hookenv.Hooks()
1996+
1997+ @hooks.hook
1998+ def config_changed():
1999+ # Print all changes to configuration from previously seen
2000+ # values.
2001+ for changed, (prev, cur) in hook_data.conf.items():
2002+ print('config changed', changed,
2003+ 'previous value', prev,
2004+ 'current value', cur)
2005+
2006+ # Get some unit specific bookeeping
2007+ if not db.get('pkg_key'):
2008+ key = urllib.urlopen('https://example.com/pkg_key').read()
2009+ db.set('pkg_key', key)
2010+
2011+ # Directly access all charm config as a mapping.
2012+ conf = db.getrange('config', True)
2013+
2014+ # Directly access all relation data as a mapping
2015+ rels = db.getrange('rels', True)
2016+
2017+ if __name__ == '__main__':
2018+ with hook_data():
2019+ hook.execute()
2020+
2021+
2022+A more basic integration is via the hook_scope context manager which simply
2023+manages transaction scope (and records hook name, and timestamp)::
2024+
2025+ >>> from unitdata import kv
2026+ >>> db = kv()
2027+ >>> with db.hook_scope('install'):
2028+ ... # do work, in transactional scope.
2029+ ... db.set('x', 1)
2030+ >>> db.get('x')
2031+ 1
2032+
2033+
2034+Usage
2035+-----
2036+
2037+Values are automatically json de/serialized to preserve basic typing
2038+and complex data struct capabilities (dicts, lists, ints, booleans, etc).
2039+
2040+Individual values can be manipulated via get/set::
2041+
2042+ >>> kv.set('y', True)
2043+ >>> kv.get('y')
2044+ True
2045+
2046+ # We can set complex values (dicts, lists) as a single key.
2047+ >>> kv.set('config', {'a': 1, 'b': True'})
2048+
2049+ # Also supports returning dictionaries as a record which
2050+ # provides attribute access.
2051+ >>> config = kv.get('config', record=True)
2052+ >>> config.b
2053+ True
2054+
2055+
2056+Groups of keys can be manipulated with update/getrange::
2057+
2058+ >>> kv.update({'z': 1, 'y': 2}, prefix="gui.")
2059+ >>> kv.getrange('gui.', strip=True)
2060+ {'z': 1, 'y': 2}
2061+
2062+When updating values, its very helpful to understand which values
2063+have actually changed and how have they changed. The storage
2064+provides a delta method to provide for this::
2065+
2066+ >>> data = {'debug': True, 'option': 2}
2067+ >>> delta = kv.delta(data, 'config.')
2068+ >>> delta.debug.previous
2069+ None
2070+ >>> delta.debug.current
2071+ True
2072+ >>> delta
2073+ {'debug': (None, True), 'option': (None, 2)}
2074+
2075+Note the delta method does not persist the actual change, it needs to
2076+be explicitly saved via 'update' method::
2077+
2078+ >>> kv.update(data, 'config.')
2079+
2080+Values modified in the context of a hook scope retain historical values
2081+associated to the hookname.
2082+
2083+ >>> with db.hook_scope('config-changed'):
2084+ ... db.set('x', 42)
2085+ >>> db.gethistory('x')
2086+ [(1, u'x', 1, u'install', u'2015-01-21T16:49:30.038372'),
2087+ (2, u'x', 42, u'config-changed', u'2015-01-21T16:49:30.038786')]
2088+
2089+"""
2090+
2091+import collections
2092+import contextlib
2093+import datetime
2094+import json
2095+import os
2096+import pprint
2097+import sqlite3
2098+import sys
2099+
2100+__author__ = 'Kapil Thangavelu <kapil.foss@gmail.com>'
2101+
2102+
2103+class Storage(object):
2104+ """Simple key value database for local unit state within charms.
2105+
2106+ Modifications are automatically committed at hook exit. That's
2107+ currently regardless of exit code.
2108+
2109+ To support dicts, lists, integer, floats, and booleans values
2110+ are automatically json encoded/decoded.
2111+ """
2112+ def __init__(self, path=None):
2113+ self.db_path = path
2114+ if path is None:
2115+ self.db_path = os.path.join(
2116+ os.environ.get('CHARM_DIR', ''), '.unit-state.db')
2117+ self.conn = sqlite3.connect('%s' % self.db_path)
2118+ self.cursor = self.conn.cursor()
2119+ self.revision = None
2120+ self._closed = False
2121+ self._init()
2122+
2123+ def close(self):
2124+ if self._closed:
2125+ return
2126+ self.flush(False)
2127+ self.cursor.close()
2128+ self.conn.close()
2129+ self._closed = True
2130+
2131+ def _scoped_query(self, stmt, params=None):
2132+ if params is None:
2133+ params = []
2134+ return stmt, params
2135+
2136+ def get(self, key, default=None, record=False):
2137+ self.cursor.execute(
2138+ *self._scoped_query(
2139+ 'select data from kv where key=?', [key]))
2140+ result = self.cursor.fetchone()
2141+ if not result:
2142+ return default
2143+ if record:
2144+ return Record(json.loads(result[0]))
2145+ return json.loads(result[0])
2146+
2147+ def getrange(self, key_prefix, strip=False):
2148+ stmt = "select key, data from kv where key like '%s%%'" % key_prefix
2149+ self.cursor.execute(*self._scoped_query(stmt))
2150+ result = self.cursor.fetchall()
2151+
2152+ if not result:
2153+ return None
2154+ if not strip:
2155+ key_prefix = ''
2156+ return dict([
2157+ (k[len(key_prefix):], json.loads(v)) for k, v in result])
2158+
2159+ def update(self, mapping, prefix=""):
2160+ for k, v in mapping.items():
2161+ self.set("%s%s" % (prefix, k), v)
2162+
2163+ def unset(self, key):
2164+ self.cursor.execute('delete from kv where key=?', [key])
2165+ if self.revision and self.cursor.rowcount:
2166+ self.cursor.execute(
2167+ 'insert into kv_revisions values (?, ?, ?)',
2168+ [key, self.revision, json.dumps('DELETED')])
2169+
2170+ def set(self, key, value):
2171+ serialized = json.dumps(value)
2172+
2173+ self.cursor.execute(
2174+ 'select data from kv where key=?', [key])
2175+ exists = self.cursor.fetchone()
2176+
2177+ # Skip mutations to the same value
2178+ if exists:
2179+ if exists[0] == serialized:
2180+ return value
2181+
2182+ if not exists:
2183+ self.cursor.execute(
2184+ 'insert into kv (key, data) values (?, ?)',
2185+ (key, serialized))
2186+ else:
2187+ self.cursor.execute('''
2188+ update kv
2189+ set data = ?
2190+ where key = ?''', [serialized, key])
2191+
2192+ # Save
2193+ if not self.revision:
2194+ return value
2195+
2196+ self.cursor.execute(
2197+ 'select 1 from kv_revisions where key=? and revision=?',
2198+ [key, self.revision])
2199+ exists = self.cursor.fetchone()
2200+
2201+ if not exists:
2202+ self.cursor.execute(
2203+ '''insert into kv_revisions (
2204+ revision, key, data) values (?, ?, ?)''',
2205+ (self.revision, key, serialized))
2206+ else:
2207+ self.cursor.execute(
2208+ '''
2209+ update kv_revisions
2210+ set data = ?
2211+ where key = ?
2212+ and revision = ?''',
2213+ [serialized, key, self.revision])
2214+
2215+ return value
2216+
2217+ def delta(self, mapping, prefix):
2218+ """
2219+ return a delta containing values that have changed.
2220+ """
2221+ previous = self.getrange(prefix, strip=True)
2222+ if not previous:
2223+ pk = set()
2224+ else:
2225+ pk = set(previous.keys())
2226+ ck = set(mapping.keys())
2227+ delta = DeltaSet()
2228+
2229+ # added
2230+ for k in ck.difference(pk):
2231+ delta[k] = Delta(None, mapping[k])
2232+
2233+ # removed
2234+ for k in pk.difference(ck):
2235+ delta[k] = Delta(previous[k], None)
2236+
2237+ # changed
2238+ for k in pk.intersection(ck):
2239+ c = mapping[k]
2240+ p = previous[k]
2241+ if c != p:
2242+ delta[k] = Delta(p, c)
2243+
2244+ return delta
2245+
2246+ @contextlib.contextmanager
2247+ def hook_scope(self, name=""):
2248+ """Scope all future interactions to the current hook execution
2249+ revision."""
2250+ assert not self.revision
2251+ self.cursor.execute(
2252+ 'insert into hooks (hook, date) values (?, ?)',
2253+ (name or sys.argv[0],
2254+ datetime.datetime.utcnow().isoformat()))
2255+ self.revision = self.cursor.lastrowid
2256+ try:
2257+ yield self.revision
2258+ self.revision = None
2259+ except:
2260+ self.flush(False)
2261+ self.revision = None
2262+ raise
2263+ else:
2264+ self.flush()
2265+
2266+ def flush(self, save=True):
2267+ if save:
2268+ self.conn.commit()
2269+ elif self._closed:
2270+ return
2271+ else:
2272+ self.conn.rollback()
2273+
2274+ def _init(self):
2275+ self.cursor.execute('''
2276+ create table if not exists kv (
2277+ key text,
2278+ data text,
2279+ primary key (key)
2280+ )''')
2281+ self.cursor.execute('''
2282+ create table if not exists kv_revisions (
2283+ key text,
2284+ revision integer,
2285+ data text,
2286+ primary key (key, revision)
2287+ )''')
2288+ self.cursor.execute('''
2289+ create table if not exists hooks (
2290+ version integer primary key autoincrement,
2291+ hook text,
2292+ date text
2293+ )''')
2294+ self.conn.commit()
2295+
2296+ def gethistory(self, key, deserialize=False):
2297+ self.cursor.execute(
2298+ '''
2299+ select kv.revision, kv.key, kv.data, h.hook, h.date
2300+ from kv_revisions kv,
2301+ hooks h
2302+ where kv.key=?
2303+ and kv.revision = h.version
2304+ ''', [key])
2305+ if deserialize is False:
2306+ return self.cursor.fetchall()
2307+ return map(_parse_history, self.cursor.fetchall())
2308+
2309+ def debug(self, fh=sys.stderr):
2310+ self.cursor.execute('select * from kv')
2311+ pprint.pprint(self.cursor.fetchall(), stream=fh)
2312+ self.cursor.execute('select * from kv_revisions')
2313+ pprint.pprint(self.cursor.fetchall(), stream=fh)
2314+
2315+
2316+def _parse_history(d):
2317+ return (d[0], d[1], json.loads(d[2]), d[3],
2318+ datetime.datetime.strptime(d[-1], "%Y-%m-%dT%H:%M:%S.%f"))
2319+
2320+
2321+class HookData(object):
2322+ """Simple integration for existing hook exec frameworks.
2323+
2324+ Records all unit information, and stores deltas for processing
2325+ by the hook.
2326+
2327+ Sample::
2328+
2329+ from charmhelper.core import hookenv, unitdata
2330+
2331+ changes = unitdata.HookData()
2332+ db = unitdata.kv()
2333+ hooks = hookenv.Hooks()
2334+
2335+ @hooks.hook
2336+ def config_changed():
2337+ # View all changes to configuration
2338+ for changed, (prev, cur) in changes.conf.items():
2339+ print('config changed', changed,
2340+ 'previous value', prev,
2341+ 'current value', cur)
2342+
2343+ # Get some unit specific bookeeping
2344+ if not db.get('pkg_key'):
2345+ key = urllib.urlopen('https://example.com/pkg_key').read()
2346+ db.set('pkg_key', key)
2347+
2348+ if __name__ == '__main__':
2349+ with changes():
2350+ hook.execute()
2351+
2352+ """
2353+ def __init__(self):
2354+ self.kv = kv()
2355+ self.conf = None
2356+ self.rels = None
2357+
2358+ @contextlib.contextmanager
2359+ def __call__(self):
2360+ from charmhelpers.core import hookenv
2361+ hook_name = hookenv.hook_name()
2362+
2363+ with self.kv.hook_scope(hook_name):
2364+ self._record_charm_version(hookenv.charm_dir())
2365+ delta_config, delta_relation = self._record_hook(hookenv)
2366+ yield self.kv, delta_config, delta_relation
2367+
2368+ def _record_charm_version(self, charm_dir):
2369+ # Record revisions.. charm revisions are meaningless
2370+ # to charm authors as they don't control the revision.
2371+ # so logic dependnent on revision is not particularly
2372+ # useful, however it is useful for debugging analysis.
2373+ charm_rev = open(
2374+ os.path.join(charm_dir, 'revision')).read().strip()
2375+ charm_rev = charm_rev or '0'
2376+ revs = self.kv.get('charm_revisions', [])
2377+ if charm_rev not in revs:
2378+ revs.append(charm_rev.strip() or '0')
2379+ self.kv.set('charm_revisions', revs)
2380+
2381+ def _record_hook(self, hookenv):
2382+ data = hookenv.execution_environment()
2383+ self.conf = conf_delta = self.kv.delta(data['conf'], 'config')
2384+ self.rels = rels_delta = self.kv.delta(data['rels'], 'rels')
2385+ self.kv.set('env', dict(data['env']))
2386+ self.kv.set('unit', data['unit'])
2387+ self.kv.set('relid', data.get('relid'))
2388+ return conf_delta, rels_delta
2389+
2390+
2391+class Record(dict):
2392+
2393+ __slots__ = ()
2394+
2395+ def __getattr__(self, k):
2396+ if k in self:
2397+ return self[k]
2398+ raise AttributeError(k)
2399+
2400+
2401+class DeltaSet(Record):
2402+
2403+ __slots__ = ()
2404+
2405+
2406+Delta = collections.namedtuple('Delta', ['previous', 'current'])
2407+
2408+
2409+_KV = None
2410+
2411+
2412+def kv():
2413+ global _KV
2414+ if _KV is None:
2415+ _KV = Storage()
2416+ return _KV
2417
2418=== modified file 'hooks/charmhelpers/fetch/archiveurl.py'
2419--- hooks/charmhelpers/fetch/archiveurl.py 2015-01-26 09:45:23 +0000
2420+++ hooks/charmhelpers/fetch/archiveurl.py 2015-04-16 21:50:07 +0000
2421@@ -18,6 +18,16 @@
2422 import hashlib
2423 import re
2424
2425+from charmhelpers.fetch import (
2426+ BaseFetchHandler,
2427+ UnhandledSource
2428+)
2429+from charmhelpers.payload.archive import (
2430+ get_archive_handler,
2431+ extract,
2432+)
2433+from charmhelpers.core.host import mkdir, check_hash
2434+
2435 import six
2436 if six.PY3:
2437 from urllib.request import (
2438@@ -35,16 +45,6 @@
2439 )
2440 from urlparse import urlparse, urlunparse, parse_qs
2441
2442-from charmhelpers.fetch import (
2443- BaseFetchHandler,
2444- UnhandledSource
2445-)
2446-from charmhelpers.payload.archive import (
2447- get_archive_handler,
2448- extract,
2449-)
2450-from charmhelpers.core.host import mkdir, check_hash
2451-
2452
2453 def splituser(host):
2454 '''urllib.splituser(), but six's support of this seems broken'''
2455
2456=== modified file 'hooks/charmhelpers/fetch/giturl.py'
2457--- hooks/charmhelpers/fetch/giturl.py 2015-01-26 09:45:23 +0000
2458+++ hooks/charmhelpers/fetch/giturl.py 2015-04-16 21:50:07 +0000
2459@@ -32,7 +32,7 @@
2460 apt_install("python-git")
2461 from git import Repo
2462
2463-from git.exc import GitCommandError
2464+from git.exc import GitCommandError # noqa E402
2465
2466
2467 class GitUrlFetchHandler(BaseFetchHandler):
2468
2469=== modified file 'hooks/glance_relations.py'
2470--- hooks/glance_relations.py 2015-01-22 16:26:28 +0000
2471+++ hooks/glance_relations.py 2015-04-16 21:50:07 +0000
2472@@ -1,18 +1,20 @@
2473 #!/usr/bin/python
2474+import sys
2475+
2476 from subprocess import (
2477+ call,
2478 check_call,
2479- call
2480 )
2481-import sys
2482
2483 from glance_utils import (
2484 do_openstack_upgrade,
2485+ git_install,
2486 migrate_database,
2487 register_configs,
2488 restart_map,
2489 services,
2490 CLUSTER_RES,
2491- PACKAGES,
2492+ determine_packages,
2493 SERVICES,
2494 CHARM,
2495 GLANCE_REGISTRY_CONF,
2496@@ -41,6 +43,7 @@
2497 )
2498 from charmhelpers.core.host import (
2499 restart_on_change,
2500+ service_reload,
2501 service_stop,
2502 )
2503 from charmhelpers.fetch import (
2504@@ -53,16 +56,19 @@
2505 get_hacluster_config
2506 )
2507 from charmhelpers.contrib.openstack.utils import (
2508+ config_value_changed,
2509 configure_installation_source,
2510- get_os_codename_package,
2511+ git_install_requested,
2512+ lsb_release,
2513 openstack_upgrade_available,
2514- lsb_release,
2515- sync_db_with_multi_ipv6_addresses
2516+ os_release,
2517+ sync_db_with_multi_ipv6_addresses,
2518 )
2519 from charmhelpers.contrib.storage.linux.ceph import (
2520 ensure_ceph_keyring,
2521 CephBrokerRq,
2522 CephBrokerRsp,
2523+ delete_keyring,
2524 )
2525 from charmhelpers.payload.execd import (
2526 execd_preinstall
2527@@ -100,7 +106,9 @@
2528 configure_installation_source(src)
2529
2530 apt_update(fatal=True)
2531- apt_install(PACKAGES, fatal=True)
2532+ apt_install(determine_packages(), fatal=True)
2533+
2534+ git_install(config('openstack-origin-git'))
2535
2536 for service in SERVICES:
2537 service_stop(service)
2538@@ -140,7 +148,7 @@
2539 @hooks.hook('shared-db-relation-changed')
2540 @restart_on_change(restart_map())
2541 def db_changed():
2542- rel = get_os_codename_package("glance-common")
2543+ rel = os_release('glance-common')
2544
2545 if 'shared-db' not in CONFIGS.complete_contexts():
2546 juju_log('shared-db relation incomplete. Peer not ready?')
2547@@ -163,7 +171,8 @@
2548 status = call(['glance-manage', 'db_version'])
2549 if status != 0:
2550 juju_log('Setting version_control to 0')
2551- check_call(["glance-manage", "version_control", "0"])
2552+ cmd = ["glance-manage", "version_control", "0"]
2553+ check_call(cmd)
2554
2555 juju_log('Cluster leader, performing db sync')
2556 migrate_database()
2557@@ -172,7 +181,7 @@
2558 @hooks.hook('pgsql-db-relation-changed')
2559 @restart_on_change(restart_map())
2560 def pgsql_db_changed():
2561- rel = get_os_codename_package("glance-common")
2562+ rel = os_release('glance-common')
2563
2564 if 'pgsql-db' not in CONFIGS.complete_contexts():
2565 juju_log('pgsql-db relation incomplete. Peer not ready?')
2566@@ -188,7 +197,8 @@
2567 status = call(['glance-manage', 'db_version'])
2568 if status != 0:
2569 juju_log('Setting version_control to 0')
2570- check_call(["glance-manage", "version_control", "0"])
2571+ cmd = ["glance-manage", "version_control", "0"]
2572+ check_call(cmd)
2573
2574 juju_log('Cluster leader, performing db sync')
2575 migrate_database()
2576@@ -263,6 +273,13 @@
2577 juju_log("Request(s) sent to Ceph broker (rid=%s)" % (rid))
2578
2579
2580+@hooks.hook('ceph-relation-broken')
2581+def ceph_broken():
2582+ service = service_name()
2583+ delete_keyring(service=service)
2584+ CONFIGS.write_all()
2585+
2586+
2587 @hooks.hook('identity-service-relation-joined')
2588 def keystone_joined(relation_id=None):
2589 public_url = '{}:9292'.format(canonical_url(CONFIGS, PUBLIC))
2590@@ -308,9 +325,13 @@
2591 sync_db_with_multi_ipv6_addresses(config('database'),
2592 config('database-user'))
2593
2594- if openstack_upgrade_available('glance-common'):
2595- juju_log('Upgrading OpenStack release')
2596- do_openstack_upgrade(CONFIGS)
2597+ if git_install_requested():
2598+ if config_value_changed('openstack-origin-git'):
2599+ git_install(config('openstack-origin-git'))
2600+ else:
2601+ if openstack_upgrade_available('glance-common'):
2602+ juju_log('Upgrading OpenStack release')
2603+ do_openstack_upgrade(CONFIGS)
2604
2605 open_port(9292)
2606 configure_https()
2607@@ -354,7 +375,7 @@
2608 @hooks.hook('upgrade-charm')
2609 @restart_on_change(restart_map(), stopstart=True)
2610 def upgrade_charm():
2611- apt_install(filter_installed_packages(PACKAGES), fatal=True)
2612+ apt_install(filter_installed_packages(determine_packages()), fatal=True)
2613 configure_https()
2614 update_nrpe_config()
2615 CONFIGS.write_all()
2616@@ -433,8 +454,7 @@
2617 [image_service_joined(rid) for rid in relation_ids('image-service')]
2618
2619
2620-@hooks.hook('ceph-relation-broken',
2621- 'identity-service-relation-broken',
2622+@hooks.hook('identity-service-relation-broken',
2623 'object-store-relation-broken',
2624 'shared-db-relation-broken',
2625 'pgsql-db-relation-broken')
2626@@ -455,6 +475,10 @@
2627 cmd = ['a2dissite', 'openstack_https_frontend']
2628 check_call(cmd)
2629
2630+ # TODO: improve this by checking if local CN certs are available
2631+ # first then checking reload status (see LP #1433114).
2632+ service_reload('apache2', restart_on_failure=True)
2633+
2634 for r_id in relation_ids('identity-service'):
2635 keystone_joined(relation_id=r_id)
2636 for r_id in relation_ids('image-service'):
2637@@ -484,7 +508,9 @@
2638 hostname = nrpe.get_nagios_hostname()
2639 current_unit = nrpe.get_nagios_unit_name()
2640 nrpe_setup = nrpe.NRPE(hostname=hostname)
2641+ nrpe.copy_nrpe_checks()
2642 nrpe.add_init_service_checks(nrpe_setup, services(), current_unit)
2643+ nrpe.add_haproxy_checks(nrpe_setup, current_unit)
2644 nrpe_setup.write()
2645
2646
2647
2648=== modified file 'hooks/glance_utils.py'
2649--- hooks/glance_utils.py 2015-01-08 10:02:48 +0000
2650+++ hooks/glance_utils.py 2015-04-16 21:50:07 +0000
2651@@ -1,6 +1,7 @@
2652 #!/usr/bin/python
2653
2654 import os
2655+import shutil
2656 import subprocess
2657
2658 import glance_contexts
2659@@ -14,21 +15,27 @@
2660 add_source)
2661
2662 from charmhelpers.core.hookenv import (
2663+ charm_dir,
2664 config,
2665 log,
2666 relation_ids,
2667 service_name)
2668
2669 from charmhelpers.core.host import (
2670+ adduser,
2671+ add_group,
2672+ add_user_to_group,
2673 mkdir,
2674 service_stop,
2675 service_start,
2676- lsb_release
2677+ service_restart,
2678+ lsb_release,
2679+ write_file,
2680 )
2681
2682 from charmhelpers.contrib.openstack import (
2683 templating,
2684- context, )
2685+ context,)
2686
2687 from charmhelpers.contrib.hahelpers.cluster import (
2688 eligible_leader,
2689@@ -37,8 +44,14 @@
2690 from charmhelpers.contrib.openstack.alternatives import install_alternative
2691 from charmhelpers.contrib.openstack.utils import (
2692 get_os_codename_install_source,
2693- get_os_codename_package,
2694- configure_installation_source)
2695+ git_install_requested,
2696+ git_clone_and_install,
2697+ git_src_dir,
2698+ configure_installation_source,
2699+ os_release,
2700+)
2701+
2702+from charmhelpers.core.templating import render
2703
2704 CLUSTER_RES = "grp_glance_vips"
2705
2706@@ -46,8 +59,27 @@
2707 "apache2", "glance", "python-mysqldb", "python-swiftclient",
2708 "python-psycopg2", "python-keystone", "python-six", "uuid", "haproxy", ]
2709
2710+BASE_GIT_PACKAGES = [
2711+ 'libxml2-dev',
2712+ 'libxslt1-dev',
2713+ 'python-dev',
2714+ 'python-pip',
2715+ 'python-setuptools',
2716+ 'zlib1g-dev',
2717+]
2718+
2719 SERVICES = [
2720- "glance-api", "glance-registry", ]
2721+ "glance-api",
2722+ "glance-registry",
2723+]
2724+
2725+# ubuntu packages that should not be installed when deploying from git
2726+GIT_PACKAGE_BLACKLIST = [
2727+ 'glance',
2728+ 'python-swiftclient',
2729+ 'python-keystone',
2730+]
2731+
2732
2733 CHARM = "glance"
2734
2735@@ -76,7 +108,9 @@
2736 (GLANCE_REGISTRY_CONF, {
2737 'hook_contexts': [context.SharedDBContext(ssl_dir=GLANCE_CONF_DIR),
2738 context.PostgresqlDBContext(),
2739- context.IdentityServiceContext(),
2740+ context.IdentityServiceContext(
2741+ service='glance',
2742+ service_user='glance'),
2743 context.SyslogContext(),
2744 glance_contexts.LoggingConfigContext(),
2745 glance_contexts.GlanceIPv6Context(),
2746@@ -90,7 +124,9 @@
2747 'hook_contexts': [context.SharedDBContext(ssl_dir=GLANCE_CONF_DIR),
2748 context.AMQPContext(ssl_dir=GLANCE_CONF_DIR),
2749 context.PostgresqlDBContext(),
2750- context.IdentityServiceContext(),
2751+ context.IdentityServiceContext(
2752+ service='glance',
2753+ service_user='glance'),
2754 glance_contexts.CephGlanceContext(),
2755 glance_contexts.ObjectStoreContext(),
2756 glance_contexts.HAProxyContext(),
2757@@ -136,7 +172,7 @@
2758 # Register config files with their respective contexts.
2759 # Regstration of some configs may not be required depending on
2760 # existing of certain relations.
2761- release = get_os_codename_package('glance-common', fatal=False) or 'essex'
2762+ release = os_release('glance-common')
2763 configs = templating.OSConfigRenderer(templates_dir=TEMPLATES,
2764 openstack_release=release)
2765
2766@@ -173,6 +209,18 @@
2767 return configs
2768
2769
2770+def determine_packages():
2771+ packages = [] + PACKAGES
2772+
2773+ if git_install_requested():
2774+ packages.extend(BASE_GIT_PACKAGES)
2775+ # don't include packages that will be installed from git
2776+ for p in GIT_PACKAGE_BLACKLIST:
2777+ packages.remove(p)
2778+
2779+ return list(set(packages))
2780+
2781+
2782 def migrate_database():
2783 '''Runs glance-manage to initialize a new database
2784 or migrate existing
2785@@ -201,7 +249,7 @@
2786 ]
2787 apt_update()
2788 apt_upgrade(options=dpkg_opts, fatal=True, dist=True)
2789- apt_install(PACKAGES, fatal=True)
2790+ apt_install(determine_packages(), fatal=True)
2791
2792 # set CONFIGS to load templates from new release and regenerate config
2793 configs.set_release(openstack_release=new_os_rel)
2794@@ -252,3 +300,85 @@
2795 ' main')
2796 apt_update()
2797 apt_install('haproxy/trusty-backports', fatal=True)
2798+
2799+
2800+def git_install(projects_yaml):
2801+ """Perform setup, and install git repos specified in yaml parameter."""
2802+ if git_install_requested():
2803+ git_pre_install()
2804+ git_clone_and_install(projects_yaml, core_project='glance')
2805+ git_post_install(projects_yaml)
2806+
2807+
2808+def git_pre_install():
2809+ """Perform glance pre-install setup."""
2810+ dirs = [
2811+ '/var/lib/glance',
2812+ '/var/lib/glance/images',
2813+ '/var/lib/glance/image-cache',
2814+ '/var/lib/glance/image-cache/incomplete',
2815+ '/var/lib/glance/image-cache/invalid',
2816+ '/var/lib/glance/image-cache/queue',
2817+ '/var/log/glance',
2818+ ]
2819+
2820+ logs = [
2821+ '/var/log/glance/glance-api.log',
2822+ '/var/log/glance/glance-registry.log',
2823+ ]
2824+
2825+ adduser('glance', shell='/bin/bash', system_user=True)
2826+ add_group('glance', system_group=True)
2827+ add_user_to_group('glance', 'glance')
2828+
2829+ for d in dirs:
2830+ mkdir(d, owner='glance', group='glance', perms=0700, force=False)
2831+
2832+ for l in logs:
2833+ write_file(l, '', owner='glance', group='glance', perms=0600)
2834+
2835+
2836+def git_post_install(projects_yaml):
2837+ """Perform glance post-install setup."""
2838+ src_etc = os.path.join(git_src_dir(projects_yaml, 'glance'), 'etc')
2839+ configs = {
2840+ 'src': src_etc,
2841+ 'dest': '/etc/glance',
2842+ }
2843+
2844+ if os.path.exists(configs['dest']):
2845+ shutil.rmtree(configs['dest'])
2846+ shutil.copytree(configs['src'], configs['dest'])
2847+
2848+ glance_api_context = {
2849+ 'service_description': 'Glance API server',
2850+ 'service_name': 'Glance',
2851+ 'user_name': 'glance',
2852+ 'start_dir': '/var/lib/glance',
2853+ 'process_name': 'glance-api',
2854+ 'executable_name': '/usr/local/bin/glance-api',
2855+ 'config_files': ['/etc/glance/glance-api.conf'],
2856+ 'log_file': '/var/log/glance/api.log',
2857+ }
2858+
2859+ glance_registry_context = {
2860+ 'service_description': 'Glance registry server',
2861+ 'service_name': 'Glance',
2862+ 'user_name': 'glance',
2863+ 'start_dir': '/var/lib/glance',
2864+ 'process_name': 'glance-registry',
2865+ 'executable_name': '/usr/local/bin/glance-registry',
2866+ 'config_files': ['/etc/glance/glance-registry.conf'],
2867+ 'log_file': '/var/log/glance/registry.log',
2868+ }
2869+
2870+ # NOTE(coreycb): Needs systemd support
2871+ templates_dir = 'hooks/charmhelpers/contrib/openstack/templates'
2872+ templates_dir = os.path.join(charm_dir(), templates_dir)
2873+ render('git.upstart', '/etc/init/glance-api.conf',
2874+ glance_api_context, perms=0o644, templates_dir=templates_dir)
2875+ render('git.upstart', '/etc/init/glance-registry.conf',
2876+ glance_registry_context, perms=0o644, templates_dir=templates_dir)
2877+
2878+ service_restart('glance-api')
2879+ service_restart('glance-registry')
2880
2881=== added directory 'templates/kilo'
2882=== added file 'templates/kilo/glance-api-paste.ini'
2883--- templates/kilo/glance-api-paste.ini 1970-01-01 00:00:00 +0000
2884+++ templates/kilo/glance-api-paste.ini 2015-04-16 21:50:07 +0000
2885@@ -0,0 +1,77 @@
2886+# Use this pipeline for no auth or image caching - DEFAULT
2887+[pipeline:glance-api]
2888+pipeline = versionnegotiation osprofiler unauthenticated-context rootapp
2889+
2890+# Use this pipeline for image caching and no auth
2891+[pipeline:glance-api-caching]
2892+pipeline = versionnegotiation osprofiler unauthenticated-context cache rootapp
2893+
2894+# Use this pipeline for caching w/ management interface but no auth
2895+[pipeline:glance-api-cachemanagement]
2896+pipeline = versionnegotiation osprofiler unauthenticated-context cache cachemanage rootapp
2897+
2898+# Use this pipeline for keystone auth
2899+[pipeline:glance-api-keystone]
2900+pipeline = versionnegotiation osprofiler authtoken context rootapp
2901+
2902+# Use this pipeline for keystone auth with image caching
2903+[pipeline:glance-api-keystone+caching]
2904+pipeline = versionnegotiation osprofiler authtoken context cache rootapp
2905+
2906+# Use this pipeline for keystone auth with caching and cache management
2907+[pipeline:glance-api-keystone+cachemanagement]
2908+pipeline = versionnegotiation osprofiler authtoken context cache cachemanage rootapp
2909+
2910+# Use this pipeline for authZ only. This means that the registry will treat a
2911+# user as authenticated without making requests to keystone to reauthenticate
2912+# the user.
2913+[pipeline:glance-api-trusted-auth]
2914+pipeline = versionnegotiation osprofiler context rootapp
2915+
2916+# Use this pipeline for authZ only. This means that the registry will treat a
2917+# user as authenticated without making requests to keystone to reauthenticate
2918+# the user and uses cache management
2919+[pipeline:glance-api-trusted-auth+cachemanagement]
2920+pipeline = versionnegotiation osprofiler context cache cachemanage rootapp
2921+
2922+[composite:rootapp]
2923+paste.composite_factory = glance.api:root_app_factory
2924+/: apiversions
2925+/v1: apiv1app
2926+/v2: apiv2app
2927+
2928+[app:apiversions]
2929+paste.app_factory = glance.api.versions:create_resource
2930+
2931+[app:apiv1app]
2932+paste.app_factory = glance.api.v1.router:API.factory
2933+
2934+[app:apiv2app]
2935+paste.app_factory = glance.api.v2.router:API.factory
2936+
2937+[filter:versionnegotiation]
2938+paste.filter_factory = glance.api.middleware.version_negotiation:VersionNegotiationFilter.factory
2939+
2940+[filter:cache]
2941+paste.filter_factory = glance.api.middleware.cache:CacheFilter.factory
2942+
2943+[filter:cachemanage]
2944+paste.filter_factory = glance.api.middleware.cache_manage:CacheManageFilter.factory
2945+
2946+[filter:context]
2947+paste.filter_factory = glance.api.middleware.context:ContextMiddleware.factory
2948+
2949+[filter:unauthenticated-context]
2950+paste.filter_factory = glance.api.middleware.context:UnauthenticatedContextMiddleware.factory
2951+
2952+[filter:authtoken]
2953+paste.filter_factory = keystonemiddleware.auth_token:filter_factory
2954+delay_auth_decision = true
2955+
2956+[filter:gzip]
2957+paste.filter_factory = glance.api.middleware.gzip:GzipMiddleware.factory
2958+
2959+[filter:osprofiler]
2960+paste.filter_factory = osprofiler.web:WsgiMiddleware.factory
2961+hmac_keys = SECRET_KEY
2962+enabled = yes
2963
2964=== added file 'templates/kilo/glance-api.conf'
2965--- templates/kilo/glance-api.conf 1970-01-01 00:00:00 +0000
2966+++ templates/kilo/glance-api.conf 2015-04-16 21:50:07 +0000
2967@@ -0,0 +1,83 @@
2968+[DEFAULT]
2969+verbose = {{ verbose }}
2970+use_syslog = {{ use_syslog }}
2971+debug = {{ debug }}
2972+workers = {{ workers }}
2973+
2974+known_stores = {{ known_stores }}
2975+{% if rbd_pool -%}
2976+default_store = rbd
2977+{% elif swift_store -%}
2978+default_store = swift
2979+{% else -%}
2980+default_store = file
2981+{% endif -%}
2982+
2983+bind_host = {{ bind_host }}
2984+
2985+{% if ext -%}
2986+bind_port = {{ ext }}
2987+{% elif bind_port -%}
2988+bind_port = {{ bind_port }}
2989+{% else -%}
2990+bind_port = 9292
2991+{% endif -%}
2992+
2993+log_file = /var/log/glance/api.log
2994+backlog = 4096
2995+
2996+registry_host = {{ registry_host }}
2997+registry_port = 9191
2998+registry_client_protocol = http
2999+
3000+{% if api_config_flags -%}
3001+{% for key, value in api_config_flags.iteritems() -%}
3002+{{ key }} = {{ value }}
3003+{% endfor -%}
3004+{% endif -%}
3005+
3006+{% if rabbitmq_host or rabbitmq_hosts -%}
3007+notification_driver = rabbit
3008+{% endif -%}
3009+
3010+{% if swift_store -%}
3011+swift_store_auth_version = 2
3012+swift_store_auth_address = {{ service_protocol }}://{{ service_host }}:{{ service_port }}/v2.0/
3013+swift_store_user = {{ admin_tenant_name }}:{{ admin_user }}
3014+swift_store_key = {{ admin_password }}
3015+swift_store_create_container_on_put = True
3016+swift_store_container = glance
3017+swift_store_large_object_size = 5120
3018+swift_store_large_object_chunk_size = 200
3019+swift_enable_snet = False
3020+{% endif -%}
3021+
3022+{% if rbd_pool -%}
3023+rbd_store_ceph_conf = /etc/ceph/ceph.conf
3024+rbd_store_user = {{ rbd_user }}
3025+rbd_store_pool = {{ rbd_pool }}
3026+rbd_store_chunk_size = 8
3027+{% endif -%}
3028+
3029+delayed_delete = False
3030+scrub_time = 43200
3031+scrubber_datadir = /var/lib/glance/scrubber
3032+image_cache_dir = /var/lib/glance/image-cache/
3033+db_enforce_mysql_charset = False
3034+
3035+[glance_store]
3036+filesystem_store_datadir = /var/lib/glance/images/
3037+
3038+[image_format]
3039+disk_formats=ami,ari,aki,vhd,vmdk,raw,qcow2,vdi,iso,root-tar
3040+
3041+{% include "section-keystone-authtoken" %}
3042+
3043+{% if auth_host -%}
3044+[paste_deploy]
3045+flavor = keystone
3046+{% endif %}
3047+
3048+{% include "parts/section-database" %}
3049+
3050+{% include "section-rabbitmq-oslo" %}
3051
3052=== added file 'templates/kilo/glance-registry-paste.ini'
3053--- templates/kilo/glance-registry-paste.ini 1970-01-01 00:00:00 +0000
3054+++ templates/kilo/glance-registry-paste.ini 2015-04-16 21:50:07 +0000
3055@@ -0,0 +1,30 @@
3056+# Use this pipeline for no auth - DEFAULT
3057+[pipeline:glance-registry]
3058+pipeline = osprofiler unauthenticated-context registryapp
3059+
3060+# Use this pipeline for keystone auth
3061+[pipeline:glance-registry-keystone]
3062+pipeline = osprofiler authtoken context registryapp
3063+
3064+# Use this pipeline for authZ only. This means that the registry will treat a
3065+# user as authenticated without making requests to keystone to reauthenticate
3066+# the user.
3067+[pipeline:glance-registry-trusted-auth]
3068+pipeline = osprofiler context registryapp
3069+
3070+[app:registryapp]
3071+paste.app_factory = glance.registry.api:API.factory
3072+
3073+[filter:context]
3074+paste.filter_factory = glance.api.middleware.context:ContextMiddleware.factory
3075+
3076+[filter:unauthenticated-context]
3077+paste.filter_factory = glance.api.middleware.context:UnauthenticatedContextMiddleware.factory
3078+
3079+[filter:authtoken]
3080+paste.filter_factory = keystonemiddleware.auth_token:filter_factory
3081+
3082+[filter:osprofiler]
3083+paste.filter_factory = osprofiler.web:WsgiMiddleware.factory
3084+hmac_keys = SECRET_KEY
3085+enabled = yes
3086
3087=== added file 'templates/kilo/glance-registry.conf'
3088--- templates/kilo/glance-registry.conf 1970-01-01 00:00:00 +0000
3089+++ templates/kilo/glance-registry.conf 2015-04-16 21:50:07 +0000
3090@@ -0,0 +1,27 @@
3091+[DEFAULT]
3092+verbose = {{ verbose }}
3093+use_syslog = {{ use_syslog }}
3094+debug = {{ debug }}
3095+workers = {{ workers }}
3096+
3097+bind_host = {{ bind_host }}
3098+bind_port = 9191
3099+log_file = /var/log/glance/registry.log
3100+backlog = 4096
3101+api_limit_max = 1000
3102+limit_param_default = 25
3103+
3104+{% if registry_config_flags -%}
3105+{% for key, value in registry_config_flags.iteritems() -%}
3106+{{ key }} = {{ value }}
3107+{% endfor -%}
3108+{% endif -%}
3109+
3110+{% include "section-keystone-authtoken" %}
3111+
3112+{% if auth_host -%}
3113+[paste_deploy]
3114+flavor = keystone
3115+{% endif %}
3116+
3117+{% include "parts/section-database" %}
3118
3119=== modified file 'templates/parts/keystone'
3120--- templates/parts/keystone 2014-04-12 16:16:54 +0000
3121+++ templates/parts/keystone 2015-04-16 21:50:07 +0000
3122@@ -7,6 +7,7 @@
3123 admin_tenant_name = {{ admin_tenant_name }}
3124 admin_user = {{ admin_user }}
3125 admin_password = {{ admin_password }}
3126+signing_dir = {{ signing_dir }}
3127
3128 [paste_deploy]
3129 flavor = keystone
3130
3131=== modified file 'templates/parts/section-database'
3132--- templates/parts/section-database 2014-04-12 15:29:10 +0000
3133+++ templates/parts/section-database 2015-04-16 21:50:07 +0000
3134@@ -1,4 +1,5 @@
3135 {% if database_host -%}
3136 [database]
3137 connection = {{ database_type }}://{{ database_user }}:{{ database_password }}@{{ database_host }}/{{ database }}{% if database_ssl_ca %}?ssl_ca={{ database_ssl_ca }}{% if database_ssl_cert %}&ssl_cert={{ database_ssl_cert }}&ssl_key={{ database_ssl_key }}{% endif %}{% endif %}
3138+idle_timeout = 3600
3139 {% endif -%}
3140
3141=== renamed file 'tests/14-basic-precise-icehouse' => 'tests/014-basic-precise-icehouse'
3142=== renamed file 'tests/15-basic-trusty-icehouse' => 'tests/015-basic-trusty-icehouse'
3143=== added file 'tests/016-basic-trusty-juno'
3144--- tests/016-basic-trusty-juno 1970-01-01 00:00:00 +0000
3145+++ tests/016-basic-trusty-juno 2015-04-16 21:50:07 +0000
3146@@ -0,0 +1,11 @@
3147+#!/usr/bin/python
3148+
3149+"""Amulet tests on a basic Glance deployment on trusty-juno."""
3150+
3151+from basic_deployment import GlanceBasicDeployment
3152+
3153+if __name__ == '__main__':
3154+ deployment = GlanceBasicDeployment(series='trusty',
3155+ openstack='cloud:trusty-juno',
3156+ source='cloud:trusty-updates/juno')
3157+ deployment.run_tests()
3158
3159=== added file 'tests/017-basic-trusty-kilo'
3160--- tests/017-basic-trusty-kilo 1970-01-01 00:00:00 +0000
3161+++ tests/017-basic-trusty-kilo 2015-04-16 21:50:07 +0000
3162@@ -0,0 +1,11 @@
3163+#!/usr/bin/python
3164+
3165+"""Amulet tests on a basic glance deployment on trusty-kilo."""
3166+
3167+from basic_deployment import GlanceBasicDeployment
3168+
3169+if __name__ == '__main__':
3170+ deployment = GlanceBasicDeployment(series='trusty',
3171+ openstack='cloud:trusty-kilo',
3172+ source='cloud:trusty-updates/kilo')
3173+ deployment.run_tests()
3174
3175=== added file 'tests/018-basic-utopic-juno'
3176--- tests/018-basic-utopic-juno 1970-01-01 00:00:00 +0000
3177+++ tests/018-basic-utopic-juno 2015-04-16 21:50:07 +0000
3178@@ -0,0 +1,9 @@
3179+#!/usr/bin/python
3180+
3181+"""Amulet tests on a basic Glance deployment on utopic-juno."""
3182+
3183+from basic_deployment import GlanceBasicDeployment
3184+
3185+if __name__ == '__main__':
3186+ deployment = GlanceBasicDeployment(series='utopic')
3187+ deployment.run_tests()
3188
3189=== added file 'tests/019-basic-vivid-kilo'
3190--- tests/019-basic-vivid-kilo 1970-01-01 00:00:00 +0000
3191+++ tests/019-basic-vivid-kilo 2015-04-16 21:50:07 +0000
3192@@ -0,0 +1,9 @@
3193+#!/usr/bin/python
3194+
3195+"""Amulet tests on a basic Glance deployment on vivid-kilo."""
3196+
3197+from basic_deployment import GlanceBasicDeployment
3198+
3199+if __name__ == '__main__':
3200+ deployment = GlanceBasicDeployment(series='vivid')
3201+ deployment.run_tests()
3202
3203=== added file 'tests/050-basic-trusty-icehouse-git'
3204--- tests/050-basic-trusty-icehouse-git 1970-01-01 00:00:00 +0000
3205+++ tests/050-basic-trusty-icehouse-git 2015-04-16 21:50:07 +0000
3206@@ -0,0 +1,9 @@
3207+#!/usr/bin/python
3208+
3209+"""Amulet tests on a basic Glance git deployment on trusty-icehouse."""
3210+
3211+from basic_deployment import GlanceBasicDeployment
3212+
3213+if __name__ == '__main__':
3214+ deployment = GlanceBasicDeployment(series='trusty', git=True)
3215+ deployment.run_tests()
3216
3217=== added file 'tests/051-basic-trusty-juno-git'
3218--- tests/051-basic-trusty-juno-git 1970-01-01 00:00:00 +0000
3219+++ tests/051-basic-trusty-juno-git 2015-04-16 21:50:07 +0000
3220@@ -0,0 +1,12 @@
3221+#!/usr/bin/python
3222+
3223+"""Amulet tests on a basic Glance git deployment on trusty-juno."""
3224+
3225+from basic_deployment import GlanceBasicDeployment
3226+
3227+if __name__ == '__main__':
3228+ deployment = GlanceBasicDeployment(series='trusty',
3229+ openstack='cloud:trusty-juno',
3230+ source='cloud:trusty-updates/juno',
3231+ git=True)
3232+ deployment.run_tests()
3233
3234=== removed file 'tests/10-basic-precise-essex'
3235--- tests/10-basic-precise-essex 2014-07-11 14:11:03 +0000
3236+++ tests/10-basic-precise-essex 1970-01-01 00:00:00 +0000
3237@@ -1,9 +0,0 @@
3238-#!/usr/bin/python
3239-
3240-"""Amulet tests on a basic glance deployment on precise-essex."""
3241-
3242-from basic_deployment import GlanceBasicDeployment
3243-
3244-if __name__ == '__main__':
3245- deployment = GlanceBasicDeployment(series='precise')
3246- deployment.run_tests()
3247
3248=== removed file 'tests/11-basic-precise-folsom'
3249--- tests/11-basic-precise-folsom 2014-07-11 14:11:03 +0000
3250+++ tests/11-basic-precise-folsom 1970-01-01 00:00:00 +0000
3251@@ -1,11 +0,0 @@
3252-#!/usr/bin/python
3253-
3254-"""Amulet tests on a basic glance deployment on precise-folsom."""
3255-
3256-from basic_deployment import GlanceBasicDeployment
3257-
3258-if __name__ == '__main__':
3259- deployment = GlanceBasicDeployment(series='precise',
3260- openstack='cloud:precise-folsom',
3261- source='cloud:precise-updates/folsom')
3262- deployment.run_tests()
3263
3264=== removed file 'tests/12-basic-precise-grizzly'
3265--- tests/12-basic-precise-grizzly 2014-07-11 14:11:03 +0000
3266+++ tests/12-basic-precise-grizzly 1970-01-01 00:00:00 +0000
3267@@ -1,11 +0,0 @@
3268-#!/usr/bin/python
3269-
3270-"""Amulet tests on a basic glance deployment on precise-grizzly."""
3271-
3272-from basic_deployment import GlanceBasicDeployment
3273-
3274-if __name__ == '__main__':
3275- deployment = GlanceBasicDeployment(series='precise',
3276- openstack='cloud:precise-grizzly',
3277- source='cloud:precise-updates/grizzly')
3278- deployment.run_tests()
3279
3280=== removed file 'tests/13-basic-precise-havana'
3281--- tests/13-basic-precise-havana 2014-07-11 14:11:03 +0000
3282+++ tests/13-basic-precise-havana 1970-01-01 00:00:00 +0000
3283@@ -1,11 +0,0 @@
3284-#!/usr/bin/python
3285-
3286-"""Amulet tests on a basic glance deployment on precise-havana."""
3287-
3288-from basic_deployment import GlanceBasicDeployment
3289-
3290-if __name__ == '__main__':
3291- deployment = GlanceBasicDeployment(series='precise',
3292- openstack='cloud:precise-havana',
3293- source='cloud:precise-updates/havana')
3294- deployment.run_tests()
3295
3296=== modified file 'tests/basic_deployment.py' (properties changed: +x to -x)
3297--- tests/basic_deployment.py 2015-04-09 00:37:53 +0000
3298+++ tests/basic_deployment.py 2015-04-16 21:50:07 +0000
3299@@ -1,6 +1,8 @@
3300 #!/usr/bin/python
3301
3302 import amulet
3303+import os
3304+import yaml
3305
3306 from charmhelpers.contrib.openstack.amulet.deployment import (
3307 OpenStackAmuletDeployment
3308@@ -13,7 +15,7 @@
3309 )
3310
3311 # Use DEBUG to turn on debug logging
3312-u = OpenStackAmuletUtils(ERROR)
3313+u = OpenStackAmuletUtils(DEBUG)
3314
3315 class GlanceBasicDeployment(OpenStackAmuletDeployment):
3316 '''Amulet tests on a basic file-backed glance deployment. Verify relations,
3317@@ -23,9 +25,11 @@
3318 # * Add tests with different storage back ends
3319 # * Resolve Essex->Havana juju set charm bug
3320
3321- def __init__(self, series=None, openstack=None, source=None, stable=True):
3322+ def __init__(self, series=None, openstack=None, source=None, git=False,
3323+ stable=False):
3324 '''Deploy the entire test environment.'''
3325 super(GlanceBasicDeployment, self).__init__(series, openstack, source, stable)
3326+ self.git = git
3327 self._add_services()
3328 self._add_relations()
3329 self._configure_services()
3330@@ -55,11 +59,30 @@
3331
3332 def _configure_services(self):
3333 '''Configure all of the services.'''
3334+ glance_config = {}
3335+ if self.git:
3336+ branch = 'stable/' + self._get_openstack_release_string()
3337+ amulet_http_proxy = os.environ.get('AMULET_HTTP_PROXY')
3338+ openstack_origin_git = {
3339+ 'repositories': [
3340+ {'name': 'requirements',
3341+ 'repository': 'git://git.openstack.org/openstack/requirements',
3342+ 'branch': branch},
3343+ {'name': 'glance',
3344+ 'repository': 'git://git.openstack.org/openstack/glance',
3345+ 'branch': branch},
3346+ ],
3347+ 'directory': '/mnt/openstack-git',
3348+ 'http_proxy': amulet_http_proxy,
3349+ 'https_proxy': amulet_http_proxy,
3350+ }
3351+ glance_config['openstack-origin-git'] = yaml.dump(openstack_origin_git)
3352+
3353 keystone_config = {'admin-password': 'openstack',
3354 'admin-token': 'ubuntutesting'}
3355-
3356 mysql_config = {'dataset-size': '50%'}
3357- configs = {'keystone': keystone_config,
3358+ configs = {'glance': glance_config,
3359+ 'keystone': keystone_config,
3360 'mysql': mysql_config}
3361 super(GlanceBasicDeployment, self)._configure_services(configs)
3362
3363
3364=== modified file 'tests/charmhelpers/contrib/amulet/utils.py'
3365--- tests/charmhelpers/contrib/amulet/utils.py 2015-01-26 09:45:23 +0000
3366+++ tests/charmhelpers/contrib/amulet/utils.py 2015-04-16 21:50:07 +0000
3367@@ -118,6 +118,9 @@
3368 longs, or can be a function that evaluate a variable and returns a
3369 bool.
3370 """
3371+ self.log.debug('actual: {}'.format(repr(actual)))
3372+ self.log.debug('expected: {}'.format(repr(expected)))
3373+
3374 for k, v in six.iteritems(expected):
3375 if k in actual:
3376 if (isinstance(v, six.string_types) or
3377@@ -134,7 +137,6 @@
3378 def validate_relation_data(self, sentry_unit, relation, expected):
3379 """Validate actual relation data based on expected relation data."""
3380 actual = sentry_unit.relation(relation[0], relation[1])
3381- self.log.debug('actual: {}'.format(repr(actual)))
3382 return self._validate_dict_data(expected, actual)
3383
3384 def _validate_list_data(self, expected, actual):
3385@@ -169,8 +171,13 @@
3386 cmd = 'pgrep -o -f {}'.format(service)
3387 else:
3388 cmd = 'pgrep -o {}'.format(service)
3389- proc_dir = '/proc/{}'.format(sentry_unit.run(cmd)[0].strip())
3390- return self._get_dir_mtime(sentry_unit, proc_dir)
3391+ cmd = cmd + ' | grep -v pgrep || exit 0'
3392+ cmd_out = sentry_unit.run(cmd)
3393+ self.log.debug('CMDout: ' + str(cmd_out))
3394+ if cmd_out[0]:
3395+ self.log.debug('Pid for %s %s' % (service, str(cmd_out[0])))
3396+ proc_dir = '/proc/{}'.format(cmd_out[0].strip())
3397+ return self._get_dir_mtime(sentry_unit, proc_dir)
3398
3399 def service_restarted(self, sentry_unit, service, filename,
3400 pgrep_full=False, sleep_time=20):
3401@@ -187,6 +194,121 @@
3402 else:
3403 return False
3404
3405+ def service_restarted_since(self, sentry_unit, mtime, service,
3406+ pgrep_full=False, sleep_time=20,
3407+ retry_count=2):
3408+ """Check if service was been started after a given time.
3409+
3410+ Args:
3411+ sentry_unit (sentry): The sentry unit to check for the service on
3412+ mtime (float): The epoch time to check against
3413+ service (string): service name to look for in process table
3414+ pgrep_full (boolean): Use full command line search mode with pgrep
3415+ sleep_time (int): Seconds to sleep before looking for process
3416+ retry_count (int): If service is not found, how many times to retry
3417+
3418+ Returns:
3419+ bool: True if service found and its start time it newer than mtime,
3420+ False if service is older than mtime or if service was
3421+ not found.
3422+ """
3423+ self.log.debug('Checking %s restarted since %s' % (service, mtime))
3424+ time.sleep(sleep_time)
3425+ proc_start_time = self._get_proc_start_time(sentry_unit, service,
3426+ pgrep_full)
3427+ while retry_count > 0 and not proc_start_time:
3428+ self.log.debug('No pid file found for service %s, will retry %i '
3429+ 'more times' % (service, retry_count))
3430+ time.sleep(30)
3431+ proc_start_time = self._get_proc_start_time(sentry_unit, service,
3432+ pgrep_full)
3433+ retry_count = retry_count - 1
3434+
3435+ if not proc_start_time:
3436+ self.log.warn('No proc start time found, assuming service did '
3437+ 'not start')
3438+ return False
3439+ if proc_start_time >= mtime:
3440+ self.log.debug('proc start time is newer than provided mtime'
3441+ '(%s >= %s)' % (proc_start_time, mtime))
3442+ return True
3443+ else:
3444+ self.log.warn('proc start time (%s) is older than provided mtime '
3445+ '(%s), service did not restart' % (proc_start_time,
3446+ mtime))
3447+ return False
3448+
3449+ def config_updated_since(self, sentry_unit, filename, mtime,
3450+ sleep_time=20):
3451+ """Check if file was modified after a given time.
3452+
3453+ Args:
3454+ sentry_unit (sentry): The sentry unit to check the file mtime on
3455+ filename (string): The file to check mtime of
3456+ mtime (float): The epoch time to check against
3457+ sleep_time (int): Seconds to sleep before looking for process
3458+
3459+ Returns:
3460+ bool: True if file was modified more recently than mtime, False if
3461+ file was modified before mtime,
3462+ """
3463+ self.log.debug('Checking %s updated since %s' % (filename, mtime))
3464+ time.sleep(sleep_time)
3465+ file_mtime = self._get_file_mtime(sentry_unit, filename)
3466+ if file_mtime >= mtime:
3467+ self.log.debug('File mtime is newer than provided mtime '
3468+ '(%s >= %s)' % (file_mtime, mtime))
3469+ return True
3470+ else:
3471+ self.log.warn('File mtime %s is older than provided mtime %s'
3472+ % (file_mtime, mtime))
3473+ return False
3474+
3475+ def validate_service_config_changed(self, sentry_unit, mtime, service,
3476+ filename, pgrep_full=False,
3477+ sleep_time=20, retry_count=2):
3478+ """Check service and file were updated after mtime
3479+
3480+ Args:
3481+ sentry_unit (sentry): The sentry unit to check for the service on
3482+ mtime (float): The epoch time to check against
3483+ service (string): service name to look for in process table
3484+ filename (string): The file to check mtime of
3485+ pgrep_full (boolean): Use full command line search mode with pgrep
3486+ sleep_time (int): Seconds to sleep before looking for process
3487+ retry_count (int): If service is not found, how many times to retry
3488+
3489+ Typical Usage:
3490+ u = OpenStackAmuletUtils(ERROR)
3491+ ...
3492+ mtime = u.get_sentry_time(self.cinder_sentry)
3493+ self.d.configure('cinder', {'verbose': 'True', 'debug': 'True'})
3494+ if not u.validate_service_config_changed(self.cinder_sentry,
3495+ mtime,
3496+ 'cinder-api',
3497+ '/etc/cinder/cinder.conf')
3498+ amulet.raise_status(amulet.FAIL, msg='update failed')
3499+ Returns:
3500+ bool: True if both service and file where updated/restarted after
3501+ mtime, False if service is older than mtime or if service was
3502+ not found or if filename was modified before mtime.
3503+ """
3504+ self.log.debug('Checking %s restarted since %s' % (service, mtime))
3505+ time.sleep(sleep_time)
3506+ service_restart = self.service_restarted_since(sentry_unit, mtime,
3507+ service,
3508+ pgrep_full=pgrep_full,
3509+ sleep_time=0,
3510+ retry_count=retry_count)
3511+ config_update = self.config_updated_since(sentry_unit, filename, mtime,
3512+ sleep_time=0)
3513+ return service_restart and config_update
3514+
3515+ def get_sentry_time(self, sentry_unit):
3516+ """Return current epoch time on a sentry"""
3517+ cmd = "date +'%s'"
3518+ return float(sentry_unit.run(cmd)[0])
3519+
3520 def relation_error(self, name, data):
3521 return 'unexpected relation data in {} - {}'.format(name, data)
3522
3523
3524=== modified file 'tests/charmhelpers/contrib/openstack/amulet/deployment.py'
3525--- tests/charmhelpers/contrib/openstack/amulet/deployment.py 2015-01-26 09:45:23 +0000
3526+++ tests/charmhelpers/contrib/openstack/amulet/deployment.py 2015-04-16 21:50:07 +0000
3527@@ -15,6 +15,7 @@
3528 # along with charm-helpers. If not, see <http://www.gnu.org/licenses/>.
3529
3530 import six
3531+from collections import OrderedDict
3532 from charmhelpers.contrib.amulet.deployment import (
3533 AmuletDeployment
3534 )
3535@@ -43,7 +44,7 @@
3536 Determine if the local branch being tested is derived from its
3537 stable or next (dev) branch, and based on this, use the corresonding
3538 stable or next branches for the other_services."""
3539- base_charms = ['mysql', 'mongodb', 'rabbitmq-server']
3540+ base_charms = ['mysql', 'mongodb']
3541
3542 if self.stable:
3543 for svc in other_services:
3544@@ -71,16 +72,19 @@
3545 services.append(this_service)
3546 use_source = ['mysql', 'mongodb', 'rabbitmq-server', 'ceph',
3547 'ceph-osd', 'ceph-radosgw']
3548+ # Openstack subordinate charms do not expose an origin option as that
3549+ # is controlled by the principle
3550+ ignore = ['neutron-openvswitch']
3551
3552 if self.openstack:
3553 for svc in services:
3554- if svc['name'] not in use_source:
3555+ if svc['name'] not in use_source + ignore:
3556 config = {'openstack-origin': self.openstack}
3557 self.d.configure(svc['name'], config)
3558
3559 if self.source:
3560 for svc in services:
3561- if svc['name'] in use_source:
3562+ if svc['name'] in use_source and svc['name'] not in ignore:
3563 config = {'source': self.source}
3564 self.d.configure(svc['name'], config)
3565
3566@@ -97,12 +101,37 @@
3567 """
3568 (self.precise_essex, self.precise_folsom, self.precise_grizzly,
3569 self.precise_havana, self.precise_icehouse,
3570- self.trusty_icehouse) = range(6)
3571+ self.trusty_icehouse, self.trusty_juno, self.trusty_kilo,
3572+ self.utopic_juno, self.vivid_kilo) = range(10)
3573 releases = {
3574 ('precise', None): self.precise_essex,
3575 ('precise', 'cloud:precise-folsom'): self.precise_folsom,
3576 ('precise', 'cloud:precise-grizzly'): self.precise_grizzly,
3577 ('precise', 'cloud:precise-havana'): self.precise_havana,
3578 ('precise', 'cloud:precise-icehouse'): self.precise_icehouse,
3579- ('trusty', None): self.trusty_icehouse}
3580+ ('trusty', None): self.trusty_icehouse,
3581+ ('trusty', 'cloud:trusty-juno'): self.trusty_juno,
3582+ ('trusty', 'cloud:trusty-kilo'): self.trusty_kilo,
3583+ ('utopic', None): self.utopic_juno,
3584+ ('vivid', None): self.vivid_kilo}
3585 return releases[(self.series, self.openstack)]
3586+
3587+ def _get_openstack_release_string(self):
3588+ """Get openstack release string.
3589+
3590+ Return a string representing the openstack release.
3591+ """
3592+ releases = OrderedDict([
3593+ ('precise', 'essex'),
3594+ ('quantal', 'folsom'),
3595+ ('raring', 'grizzly'),
3596+ ('saucy', 'havana'),
3597+ ('trusty', 'icehouse'),
3598+ ('utopic', 'juno'),
3599+ ('vivid', 'kilo'),
3600+ ])
3601+ if self.openstack:
3602+ os_origin = self.openstack.split(':')[1]
3603+ return os_origin.split('%s-' % self.series)[1].split('/')[0]
3604+ else:
3605+ return releases[self.series]
3606
3607=== added directory 'trusty'
3608=== modified file 'unit_tests/__init__.py'
3609--- unit_tests/__init__.py 2013-08-09 18:45:02 +0000
3610+++ unit_tests/__init__.py 2015-04-16 21:50:07 +0000
3611@@ -1,3 +1,4 @@
3612 import sys
3613
3614+sys.path.append('actions/')
3615 sys.path.append('hooks/')
3616
3617=== added file 'unit_tests/test_actions_git_reinstall.py'
3618--- unit_tests/test_actions_git_reinstall.py 1970-01-01 00:00:00 +0000
3619+++ unit_tests/test_actions_git_reinstall.py 2015-04-16 21:50:07 +0000
3620@@ -0,0 +1,96 @@
3621+from mock import patch
3622+import os
3623+
3624+os.environ['JUJU_UNIT_NAME'] = 'glance'
3625+
3626+with patch('glance_utils.register_configs') as register_configs:
3627+ import git_reinstall
3628+
3629+from test_utils import (
3630+ CharmTestCase
3631+)
3632+
3633+TO_PATCH = [
3634+ 'config',
3635+]
3636+
3637+
3638+openstack_origin_git = \
3639+ """repositories:
3640+ - {name: requirements,
3641+ repository: 'git://git.openstack.org/openstack/requirements',
3642+ branch: stable/juno}
3643+ - {name: glance,
3644+ repository: 'git://git.openstack.org/openstack/glance',
3645+ branch: stable/juno}"""
3646+
3647+
3648+class TestGlanceActions(CharmTestCase):
3649+
3650+ def setUp(self):
3651+ super(TestGlanceActions, self).setUp(git_reinstall, TO_PATCH)
3652+ self.config.side_effect = self.test_config.get
3653+
3654+ @patch.object(git_reinstall, 'action_set')
3655+ @patch.object(git_reinstall, 'action_fail')
3656+ @patch.object(git_reinstall, 'git_install')
3657+ @patch.object(git_reinstall, 'config_changed')
3658+ @patch('charmhelpers.contrib.openstack.utils.config')
3659+ def test_git_reinstall(self, _config, config_changed, git_install,
3660+ action_fail, action_set):
3661+ _config.return_value = openstack_origin_git
3662+ self.test_config.set('openstack-origin-git', openstack_origin_git)
3663+
3664+ git_reinstall.git_reinstall()
3665+
3666+ git_install.assert_called_with(openstack_origin_git)
3667+ self.assertTrue(git_install.called)
3668+ self.assertTrue(config_changed.called)
3669+ self.assertFalse(action_set.called)
3670+ self.assertFalse(action_fail.called)
3671+
3672+ @patch.object(git_reinstall, 'action_set')
3673+ @patch.object(git_reinstall, 'action_fail')
3674+ @patch.object(git_reinstall, 'git_install')
3675+ @patch.object(git_reinstall, 'config_changed')
3676+ @patch('charmhelpers.contrib.openstack.utils.config')
3677+ def test_git_reinstall_not_configured(self, _config, config_changed,
3678+ git_install, action_fail,
3679+ action_set):
3680+ _config.return_value = None
3681+
3682+ git_reinstall.git_reinstall()
3683+
3684+ msg = 'openstack-origin-git is not configured'
3685+ action_fail.assert_called_with(msg)
3686+ self.assertFalse(git_install.called)
3687+ self.assertFalse(action_set.called)
3688+
3689+ @patch.object(git_reinstall, 'action_set')
3690+ @patch.object(git_reinstall, 'action_fail')
3691+ @patch.object(git_reinstall, 'git_install')
3692+ @patch.object(git_reinstall, 'config_changed')
3693+ @patch('traceback.format_exc')
3694+ @patch('charmhelpers.contrib.openstack.utils.config')
3695+ def test_git_reinstall_exception(self, _config, format_exc,
3696+ config_changed, git_install, action_fail,
3697+ action_set):
3698+ _config.return_value = openstack_origin_git
3699+ e = OSError('something bad happened')
3700+ git_install.side_effect = e
3701+ traceback = (
3702+ "Traceback (most recent call last):\n"
3703+ " File \"actions/git_reinstall.py\", line 37, in git_reinstall\n"
3704+ " git_install(config(\'openstack-origin-git\'))\n"
3705+ " File \"/usr/lib/python2.7/dist-packages/mock.py\", line 964, in __call__\n" # noqa
3706+ " return _mock_self._mock_call(*args, **kwargs)\n"
3707+ " File \"/usr/lib/python2.7/dist-packages/mock.py\", line 1019, in _mock_call\n" # noqa
3708+ " raise effect\n"
3709+ "OSError: something bad happened\n")
3710+ format_exc.return_value = traceback
3711+
3712+ git_reinstall.git_reinstall()
3713+
3714+ msg = 'git-reinstall resulted in an unexpected error'
3715+ action_fail.assert_called_with(msg)
3716+ action_set.assert_called_with({'traceback': traceback})
3717
3718=== modified file 'unit_tests/test_glance_relations.py'
3719--- unit_tests/test_glance_relations.py 2015-01-22 16:55:57 +0000
3720+++ unit_tests/test_glance_relations.py 2015-04-16 21:50:07 +0000
3721@@ -1,6 +1,7 @@
3722 from mock import call, patch, MagicMock
3723 import json
3724 import os
3725+import yaml
3726
3727 from test_utils import CharmTestCase
3728
3729@@ -38,10 +39,11 @@
3730 'apt_install',
3731 'apt_update',
3732 'restart_on_change',
3733+ 'service_reload',
3734 'service_stop',
3735 # charmhelpers.contrib.openstack.utils
3736 'configure_installation_source',
3737- 'get_os_codename_package',
3738+ 'os_release',
3739 'openstack_upgrade_available',
3740 # charmhelpers.contrib.hahelpers.cluster_utils
3741 'eligible_leader',
3742@@ -52,6 +54,7 @@
3743 'migrate_database',
3744 'ensure_ceph_keyring',
3745 'ceph_config_file',
3746+ 'git_install',
3747 'update_nrpe_config',
3748 # other
3749 'call',
3750@@ -64,6 +67,7 @@
3751 'get_iface_for_address',
3752 'get_ipv6_addr',
3753 'sync_db_with_multi_ipv6_addresses',
3754+ 'delete_keyring',
3755 ]
3756
3757
3758@@ -73,23 +77,26 @@
3759 super(GlanceRelationTests, self).setUp(relations, TO_PATCH)
3760 self.config.side_effect = self.test_config.get
3761
3762- def test_install_hook(self):
3763+ @patch.object(utils, 'git_install_requested')
3764+ def test_install_hook(self, git_requested):
3765+ git_requested.return_value = False
3766 repo = 'cloud:precise-grizzly'
3767 self.test_config.set('openstack-origin', repo)
3768 self.service_stop.return_value = True
3769 relations.install_hook()
3770 self.configure_installation_source.assert_called_with(repo)
3771 self.apt_update.assert_called_with(fatal=True)
3772- self.apt_install.assert_called_with(['apache2', 'glance',
3773- 'python-mysqldb',
3774- 'python-swiftclient',
3775- 'python-psycopg2',
3776+ self.apt_install.assert_called_with(['haproxy', 'python-six', 'uuid',
3777+ 'python-mysqldb', 'apache2',
3778+ 'python-psycopg2', 'glance',
3779 'python-keystone',
3780- 'python-six',
3781- 'uuid', 'haproxy'], fatal=True)
3782+ 'python-swiftclient'], fatal=True)
3783 self.assertTrue(self.execd_preinstall.called)
3784+ self.git_install.assert_called_with(None)
3785
3786- def test_install_hook_precise_distro(self):
3787+ @patch.object(utils, 'git_install_requested')
3788+ def test_install_hook_precise_distro(self, git_requested):
3789+ git_requested.return_value = False
3790 self.test_config.set('openstack-origin', 'distro')
3791 self.lsb_release.return_value = {'DISTRIB_RELEASE': 12.04,
3792 'DISTRIB_CODENAME': 'precise'}
3793@@ -99,6 +106,37 @@
3794 "cloud:precise-folsom"
3795 )
3796
3797+ @patch.object(utils, 'git_install_requested')
3798+ def test_install_hook_git(self, git_requested):
3799+ git_requested.return_value = True
3800+ repo = 'cloud:trusty-juno'
3801+ openstack_origin_git = {
3802+ 'repositories': [
3803+ {'name': 'requirements',
3804+ 'repository': 'git://git.openstack.org/openstack/requirements', # noqa
3805+ 'branch': 'stable/juno'},
3806+ {'name': 'glance',
3807+ 'repository': 'git://git.openstack.org/openstack/glance',
3808+ 'branch': 'stable/juno'}
3809+ ],
3810+ 'directory': '/mnt/openstack-git',
3811+ }
3812+ projects_yaml = yaml.dump(openstack_origin_git)
3813+ self.test_config.set('openstack-origin', repo)
3814+ self.test_config.set('openstack-origin-git', projects_yaml)
3815+ relations.install_hook()
3816+ self.assertTrue(self.execd_preinstall.called)
3817+ self.configure_installation_source.assert_called_with(repo)
3818+ self.apt_update.assert_called_with(fatal=True)
3819+ self.apt_install.assert_called_with(['haproxy', 'python-setuptools',
3820+ 'python-six', 'uuid',
3821+ 'python-mysqldb', 'python-pip',
3822+ 'apache2', 'libxslt1-dev',
3823+ 'python-psycopg2', 'zlib1g-dev',
3824+ 'python-dev', 'libxml2-dev'],
3825+ fatal=True)
3826+ self.git_install.assert_called_with(projects_yaml)
3827+
3828 def test_db_joined(self):
3829 self.unit_get.return_value = 'glance.foohost.com'
3830 self.is_relation_made.return_value = False
3831@@ -215,7 +253,7 @@
3832
3833 @patch.object(relations, 'CONFIGS')
3834 def test_db_changed_with_essex_not_setting_version_control(self, configs):
3835- self.get_os_codename_package.return_value = "essex"
3836+ self.os_release.return_value = "essex"
3837 self.call.return_value = 0
3838 self._shared_db_test(configs, 'glance/0')
3839 self.assertEquals([call('/etc/glance/glance-registry.conf')],
3840@@ -228,7 +266,7 @@
3841 @patch.object(relations, 'CONFIGS')
3842 def test_postgresql_db_changed_with_essex_not_setting_version_control(
3843 self, configs):
3844- self.get_os_codename_package.return_value = "essex"
3845+ self.os_release.return_value = "essex"
3846 self.call.return_value = 0
3847 self._postgresql_db_test(configs)
3848 self.assertEquals([call('/etc/glance/glance-registry.conf')],
3849@@ -240,7 +278,7 @@
3850
3851 @patch.object(relations, 'CONFIGS')
3852 def test_db_changed_with_essex_setting_version_control(self, configs):
3853- self.get_os_codename_package.return_value = "essex"
3854+ self.os_release.return_value = "essex"
3855 self.call.return_value = 1
3856 self._shared_db_test(configs, 'glance/0')
3857 self.assertEquals([call('/etc/glance/glance-registry.conf')],
3858@@ -256,7 +294,7 @@
3859 @patch.object(relations, 'CONFIGS')
3860 def test_postgresql_db_changed_with_essex_setting_version_control(
3861 self, configs):
3862- self.get_os_codename_package.return_value = "essex"
3863+ self.os_release.return_value = "essex"
3864 self.call.return_value = 1
3865 self._postgresql_db_test(configs)
3866 self.assertEquals([call('/etc/glance/glance-registry.conf')],
3867@@ -381,6 +419,13 @@
3868 call(self.ceph_config_file())],
3869 configs.write.call_args_list)
3870
3871+ @patch.object(relations, 'CONFIGS')
3872+ def test_ceph_broken(self, configs):
3873+ self.service_name.return_value = 'glance'
3874+ relations.ceph_broken()
3875+ self.delete_keyring.assert_called_with(service='glance')
3876+ self.assertTrue(configs.write_all.called)
3877+
3878 def test_keystone_joined(self):
3879 self.canonical_url.return_value = 'http://glancehost'
3880 relations.keystone_joined()
3881@@ -433,8 +478,12 @@
3882 @patch.object(relations, 'configure_https')
3883 @patch.object(relations, 'object_store_joined')
3884 @patch.object(relations, 'CONFIGS')
3885- def test_keystone_changed_with_object_store_relation(
3886- self, configs, object_store_joined, configure_https):
3887+ @patch.object(utils, 'git_install_requested')
3888+ def test_keystone_changed_with_object_store_relation(self, git_requested,
3889+ configs,
3890+ object_store_joined,
3891+ configure_https):
3892+ git_requested.return_value = False
3893 configs.complete_contexts = MagicMock()
3894 configs.complete_contexts.return_value = ['identity-service']
3895 configs.write = MagicMock()
3896@@ -449,14 +498,20 @@
3897 self.assertTrue(configure_https.called)
3898
3899 @patch.object(relations, 'configure_https')
3900- def test_config_changed_no_openstack_upgrade(self, configure_https):
3901+ @patch.object(relations, 'git_install_requested')
3902+ def test_config_changed_no_openstack_upgrade(self, git_requested,
3903+ configure_https):
3904+ git_requested.return_value = False
3905 self.openstack_upgrade_available.return_value = False
3906 relations.config_changed()
3907 self.open_port.assert_called_with(9292)
3908 self.assertTrue(configure_https.called)
3909
3910 @patch.object(relations, 'configure_https')
3911- def test_config_changed_with_openstack_upgrade(self, configure_https):
3912+ @patch.object(relations, 'git_install_requested')
3913+ def test_config_changed_with_openstack_upgrade(self, git_requested,
3914+ configure_https):
3915+ git_requested.return_value = False
3916 self.openstack_upgrade_available.return_value = True
3917 relations.config_changed()
3918 self.juju_log.assert_called_with(
3919@@ -465,6 +520,32 @@
3920 self.assertTrue(self.do_openstack_upgrade.called)
3921 self.assertTrue(configure_https.called)
3922
3923+ @patch.object(relations, 'configure_https')
3924+ @patch.object(relations, 'git_install_requested')
3925+ @patch.object(relations, 'config_value_changed')
3926+ def test_config_changed_git_updated(self, config_val_changed,
3927+ git_requested, configure_https):
3928+ git_requested.return_value = True
3929+ repo = 'cloud:trusty-juno'
3930+ openstack_origin_git = {
3931+ 'repositories': [
3932+ {'name': 'requirements',
3933+ 'repository': 'git://git.openstack.org/openstack/requirements', # noqa
3934+ 'branch': 'stable/juno'},
3935+ {'name': 'glance',
3936+ 'repository': 'git://git.openstack.org/openstack/glance',
3937+ 'branch': 'stable/juno'}
3938+ ],
3939+ 'directory': '/mnt/openstack-git',
3940+ }
3941+ projects_yaml = yaml.dump(openstack_origin_git)
3942+ self.test_config.set('openstack-origin', repo)
3943+ self.test_config.set('openstack-origin-git', projects_yaml)
3944+ relations.config_changed()
3945+ self.git_install.assert_called_with(projects_yaml)
3946+ self.assertFalse(self.do_openstack_upgrade.called)
3947+ self.assertTrue(configure_https.called)
3948+
3949 @patch.object(relations, 'CONFIGS')
3950 def test_cluster_changed(self, configs):
3951 self.test_config.set('prefer-ipv6', False)
3952@@ -491,7 +572,9 @@
3953 configs.write.call_args_list)
3954
3955 @patch.object(relations, 'CONFIGS')
3956- def test_upgrade_charm(self, configs):
3957+ @patch.object(utils, 'git_install_requested')
3958+ def test_upgrade_charm(self, git_requested, configs):
3959+ git_requested.return_value = False
3960 self.filter_installed_packages.return_value = ['test']
3961 relations.upgrade_charm()
3962 self.apt_install.assert_called_with(['test'], fatal=True)
3963@@ -596,8 +679,9 @@
3964 configs.write = MagicMock()
3965 self.relation_ids.return_value = ['identity-service:0']
3966 relations.configure_https()
3967- cmd = ['a2ensite', 'openstack_https_frontend']
3968- self.check_call.assert_called_with(cmd)
3969+ calls = [call('a2dissite', 'openstack_https_frontend'),
3970+ call('service', 'apache2', 'reload')]
3971+ self.check_call.assert_called_has_calls(calls)
3972 keystone_joined.assert_called_with(relation_id='identity-service:0')
3973
3974 @patch.object(relations, 'keystone_joined')
3975@@ -609,8 +693,9 @@
3976 configs.write = MagicMock()
3977 self.relation_ids.return_value = ['identity-service:0']
3978 relations.configure_https()
3979- cmd = ['a2dissite', 'openstack_https_frontend']
3980- self.check_call.assert_called_with(cmd)
3981+ calls = [call('a2dissite', 'openstack_https_frontend'),
3982+ call('service', 'apache2', 'reload')]
3983+ self.check_call.assert_called_has_calls(calls)
3984 keystone_joined.assert_called_with(relation_id='identity-service:0')
3985
3986 @patch.object(relations, 'image_service_joined')
3987@@ -622,8 +707,9 @@
3988 configs.write = MagicMock()
3989 self.relation_ids.return_value = ['image-service:0']
3990 relations.configure_https()
3991- cmd = ['a2ensite', 'openstack_https_frontend']
3992- self.check_call.assert_called_with(cmd)
3993+ calls = [call('a2dissite', 'openstack_https_frontend'),
3994+ call('service', 'apache2', 'reload')]
3995+ self.check_call.assert_called_has_calls(calls)
3996 image_service_joined.assert_called_with(relation_id='image-service:0')
3997
3998 @patch.object(relations, 'image_service_joined')
3999@@ -635,8 +721,9 @@
4000 configs.write = MagicMock()
4001 self.relation_ids.return_value = ['image-service:0']
4002 relations.configure_https()
4003- cmd = ['a2dissite', 'openstack_https_frontend']
4004- self.check_call.assert_called_with(cmd)
4005+ calls = [call('a2dissite', 'openstack_https_frontend'),
4006+ call('service', 'apache2', 'reload')]
4007+ self.check_call.assert_called_has_calls(calls)
4008 image_service_joined.assert_called_with(relation_id='image-service:0')
4009
4010 def test_amqp_joined(self):
4011
4012=== modified file 'unit_tests/test_glance_utils.py'
4013--- unit_tests/test_glance_utils.py 2015-01-06 15:09:51 +0000
4014+++ unit_tests/test_glance_utils.py 2015-04-16 21:50:07 +0000
4015@@ -14,7 +14,6 @@
4016 'config',
4017 'log',
4018 'relation_ids',
4019- 'get_os_codename_package',
4020 'get_os_codename_install_source',
4021 'configure_installation_source',
4022 'eligible_leader',
4023@@ -23,6 +22,7 @@
4024 'apt_upgrade',
4025 'apt_install',
4026 'mkdir',
4027+ 'os_release',
4028 'service_start',
4029 'service_stop',
4030 'service_name',
4031@@ -34,6 +34,15 @@
4032 '--option', 'Dpkg::Options::=--force-confdef',
4033 ]
4034
4035+openstack_origin_git = \
4036+ """repositories:
4037+ - {name: requirements,
4038+ repository: 'git://git.openstack.org/openstack/requirements',
4039+ branch: stable/juno}
4040+ - {name: glance,
4041+ repository: 'git://git.openstack.org/openstack/glance',
4042+ branch: stable/juno}"""
4043+
4044
4045 class TestGlanceUtils(CharmTestCase):
4046
4047@@ -50,7 +59,7 @@
4048 @patch('os.path.exists')
4049 def test_register_configs_apache(self, exists):
4050 exists.return_value = False
4051- self.get_os_codename_package.return_value = 'grizzly'
4052+ self.os_release.return_value = 'grizzly'
4053 self.relation_ids.return_value = False
4054 configs = utils.register_configs()
4055 calls = []
4056@@ -69,7 +78,7 @@
4057 @patch('os.path.exists')
4058 def test_register_configs_apache24(self, exists):
4059 exists.return_value = True
4060- self.get_os_codename_package.return_value = 'grizzly'
4061+ self.os_release.return_value = 'grizzly'
4062 self.relation_ids.return_value = False
4063 configs = utils.register_configs()
4064 calls = []
4065@@ -88,7 +97,7 @@
4066 @patch('os.path.exists')
4067 def test_register_configs_ceph(self, exists):
4068 exists.return_value = True
4069- self.get_os_codename_package.return_value = 'grizzly'
4070+ self.os_release.return_value = 'grizzly'
4071 self.relation_ids.return_value = ['ceph:0']
4072 self.service_name.return_value = 'glance'
4073 configs = utils.register_configs()
4074@@ -121,8 +130,26 @@
4075 ])
4076 self.assertEquals(ex_map, utils.restart_map())
4077
4078+ @patch('charmhelpers.contrib.openstack.utils.config')
4079+ def test_determine_packages(self, _config):
4080+ _config.return_value = None
4081+ result = utils.determine_packages()
4082+ ex = utils.PACKAGES
4083+ self.assertEquals(set(ex), set(result))
4084+
4085+ @patch('charmhelpers.contrib.openstack.utils.config')
4086+ def test_determine_packages_git(self, _config):
4087+ _config.return_value = openstack_origin_git
4088+ result = utils.determine_packages()
4089+ ex = utils.PACKAGES + utils.BASE_GIT_PACKAGES
4090+ for p in utils.GIT_PACKAGE_BLACKLIST:
4091+ ex.remove(p)
4092+ self.assertEquals(set(ex), set(result))
4093+
4094 @patch.object(utils, 'migrate_database')
4095- def test_openstack_upgrade_leader(self, migrate):
4096+ @patch.object(utils, 'git_install_requested')
4097+ def test_openstack_upgrade_leader(self, git_requested, migrate):
4098+ git_requested.return_value = True
4099 self.config.side_effect = None
4100 self.config.return_value = 'cloud:precise-havana'
4101 self.eligible_leader.return_value = True
4102@@ -130,14 +157,17 @@
4103 configs = MagicMock()
4104 utils.do_openstack_upgrade(configs)
4105 self.assertTrue(configs.write_all.called)
4106- self.apt_install.assert_called_with(utils.PACKAGES, fatal=True)
4107+ self.apt_install.assert_called_with(utils.determine_packages(),
4108+ fatal=True)
4109 self.apt_upgrade.assert_called_with(options=DPKG_OPTS,
4110 fatal=True, dist=True)
4111 configs.set_release.assert_called_with(openstack_release='havana')
4112 self.assertTrue(migrate.called)
4113
4114 @patch.object(utils, 'migrate_database')
4115- def test_openstack_upgrade_not_leader(self, migrate):
4116+ @patch.object(utils, 'git_install_requested')
4117+ def test_openstack_upgrade_not_leader(self, git_requested, migrate):
4118+ git_requested.return_value = True
4119 self.config.side_effect = None
4120 self.config.return_value = 'cloud:precise-havana'
4121 self.eligible_leader.return_value = False
4122@@ -145,8 +175,111 @@
4123 configs = MagicMock()
4124 utils.do_openstack_upgrade(configs)
4125 self.assertTrue(configs.write_all.called)
4126- self.apt_install.assert_called_with(utils.PACKAGES, fatal=True)
4127+ self.apt_install.assert_called_with(utils.determine_packages(),
4128+ fatal=True)
4129 self.apt_upgrade.assert_called_with(options=DPKG_OPTS,
4130 fatal=True, dist=True)
4131 configs.set_release.assert_called_with(openstack_release='havana')
4132 self.assertFalse(migrate.called)
4133+
4134+ @patch.object(utils, 'git_install_requested')
4135+ @patch.object(utils, 'git_clone_and_install')
4136+ @patch.object(utils, 'git_post_install')
4137+ @patch.object(utils, 'git_pre_install')
4138+ def test_git_install(self, git_pre, git_post, git_clone_and_install,
4139+ git_requested):
4140+ projects_yaml = openstack_origin_git
4141+ git_requested.return_value = True
4142+ utils.git_install(projects_yaml)
4143+ self.assertTrue(git_pre.called)
4144+ git_clone_and_install.assert_called_with(openstack_origin_git,
4145+ core_project='glance')
4146+ self.assertTrue(git_post.called)
4147+
4148+ @patch.object(utils, 'mkdir')
4149+ @patch.object(utils, 'write_file')
4150+ @patch.object(utils, 'add_user_to_group')
4151+ @patch.object(utils, 'add_group')
4152+ @patch.object(utils, 'adduser')
4153+ def test_git_pre_install(self, adduser, add_group, add_user_to_group,
4154+ write_file, mkdir):
4155+ utils.git_pre_install()
4156+ adduser.assert_called_with('glance', shell='/bin/bash',
4157+ system_user=True)
4158+ add_group.assert_called_with('glance', system_group=True)
4159+ add_user_to_group.assert_called_with('glance', 'glance')
4160+ expected = [
4161+ call('/var/lib/glance', owner='glance',
4162+ group='glance', perms=0700, force=False),
4163+ call('/var/lib/glance/images', owner='glance',
4164+ group='glance', perms=0700, force=False),
4165+ call('/var/lib/glance/image-cache', owner='glance',
4166+ group='glance', perms=0700, force=False),
4167+ call('/var/lib/glance/image-cache/incomplete', owner='glance',
4168+ group='glance', perms=0700, force=False),
4169+ call('/var/lib/glance/image-cache/invalid', owner='glance',
4170+ group='glance', perms=0700, force=False),
4171+ call('/var/lib/glance/image-cache/queue', owner='glance',
4172+ group='glance', perms=0700, force=False),
4173+ call('/var/log/glance', owner='glance',
4174+ group='glance', perms=0700, force=False),
4175+ ]
4176+ self.assertEquals(mkdir.call_args_list, expected)
4177+ expected = [
4178+ call('/var/log/glance/glance-api.log', '', owner='glance',
4179+ group='glance', perms=0600),
4180+ call('/var/log/glance/glance-registry.log', '', owner='glance',
4181+ group='glance', perms=0600),
4182+ ]
4183+ self.assertEquals(write_file.call_args_list, expected)
4184+
4185+ @patch.object(utils, 'git_src_dir')
4186+ @patch.object(utils, 'service_restart')
4187+ @patch.object(utils, 'render')
4188+ @patch('os.path.join')
4189+ @patch('os.path.exists')
4190+ @patch('shutil.copytree')
4191+ @patch('shutil.rmtree')
4192+ def test_git_post_install(self, rmtree, copytree, exists, join, render,
4193+ service_restart, git_src_dir):
4194+ projects_yaml = openstack_origin_git
4195+ join.return_value = 'joined-string'
4196+ utils.git_post_install(projects_yaml)
4197+ expected = [
4198+ call('joined-string', '/etc/glance'),
4199+ ]
4200+ copytree.assert_has_calls(expected)
4201+ glance_api_context = {
4202+ 'service_description': 'Glance API server',
4203+ 'service_name': 'Glance',
4204+ 'user_name': 'glance',
4205+ 'start_dir': '/var/lib/glance',
4206+ 'process_name': 'glance-api',
4207+ 'executable_name': '/usr/local/bin/glance-api',
4208+ 'config_files': ['/etc/glance/glance-api.conf'],
4209+ 'log_file': '/var/log/glance/api.log',
4210+ }
4211+ glance_registry_context = {
4212+ 'service_description': 'Glance registry server',
4213+ 'service_name': 'Glance',
4214+ 'user_name': 'glance',
4215+ 'start_dir': '/var/lib/glance',
4216+ 'process_name': 'glance-registry',
4217+ 'executable_name': '/usr/local/bin/glance-registry',
4218+ 'config_files': ['/etc/glance/glance-registry.conf'],
4219+ 'log_file': '/var/log/glance/registry.log',
4220+ }
4221+ expected = [
4222+ call('git.upstart', '/etc/init/glance-api.conf',
4223+ glance_api_context, perms=0o644,
4224+ templates_dir='joined-string'),
4225+ call('git.upstart', '/etc/init/glance-registry.conf',
4226+ glance_registry_context, perms=0o644,
4227+ templates_dir='joined-string'),
4228+ ]
4229+ self.assertEquals(render.call_args_list, expected)
4230+ expected = [
4231+ call('glance-api'),
4232+ call('glance-registry'),
4233+ ]
4234+ self.assertEquals(service_restart.call_args_list, expected)

Subscribers

People subscribed via source and target branches