Merge lp:~cbjchen/charms/trusty/heat/next-ha into lp:~openstack-charmers-archive/charms/trusty/heat/next

[Liang Chen]

Provide Heat charm HA support

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #15610 heat-next for cbjchen mp280946
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/14086903/
Build: http://10.245.162.77:8080/job/charm_lint_check/15610/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #14567 heat-next for cbjchen mp280946
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/14086905/
Build: http://10.245.162.77:8080/job/charm_unit_test/14567/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8330 heat-next for cbjchen mp280946
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14086964/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8330/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #16509 heat-next for cbjchen mp280946
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/16509/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #15415 heat-next for cbjchen mp280946
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: pastebin not avail., cmd error
Build: http://10.245.162.77:8080/job/charm_unit_test/15415/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8495 heat-next for cbjchen mp280946
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14399097/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8495/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #15481 heat-next for cbjchen mp280946
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/14407592/
Build: http://10.245.162.77:8080/job/charm_unit_test/15481/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #16585 heat-next for cbjchen mp280946
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/16585/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8502 heat-next for cbjchen mp280946
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14407692/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8502/

[Liam Young (gnuoy)]

amulet tests and unit tests need some tweaks. Otherwise looks good to me, thanks for mp!

review: Needs Fixing

[Liang Chen (cbjchen)]

rebased onto the latest heat/next branch

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #16835 heat-next for cbjchen mp280946
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
make: *** [lint] Error 1
ERROR:root:Make target returned non-zero.

Full lint test output: http://paste.ubuntu.com/14438179/
Build: http://10.245.162.77:8080/job/charm_lint_check/16835/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #15723 heat-next for cbjchen mp280946
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/15723/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #15842 heat-next for cbjchen mp280946
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
make: *** [test] Error 2
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/14445600/
Build: http://10.245.162.77:8080/job/charm_unit_test/15842/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #16962 heat-next for cbjchen mp280946
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/16962/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8612 heat-next for cbjchen mp280946
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/8612/

[Liang Chen (cbjchen)]

Hi Liam,
Patch rebased onto the latest heat/next branch.
amulet tests and unit tests are fixed.

Thanks,
Liang

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #16963 heat-next for cbjchen mp280946
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/16963/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #15843 heat-next for cbjchen mp280946
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/15843/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8613 heat-next for cbjchen mp280946
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/14447806/
Build: http://10.245.162.77:8080/job/charm_amulet_test/8613/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #8617 heat-next for cbjchen mp280946
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/8617/

[lp:~cbjchen/charms/trusty/heat/next-ha]

66. By Liam Young on 2016-01-12

Update test combo definitions, remove Vivid deprecated release tests, update bundletester testplan yaml, update tests README.

67. By James Page on 2016-01-19

Fix liberty/mitaka typo from previous test definition update batch.

68. By James Page on 2016-02-10

Fix support for liberty and kilo openstack releases.

69. By James Page on 2016-02-16

Tidy tox targets

70. By Edward Hope-Morley on 2016-02-18

[hopem,r=jamespage]

Sync charmhelpers to get support for supplying a Ascii Armor PGP key to
openstack-origin (currently only supports Radix64)
Closes-Bug: 1518975

71. By Liang Chen on 2016-02-19

Provide heat charm HA support

[cbjchen,r=]

Unmerged revisions

=== modified file 'config.yaml'
--- config.yaml	2015-12-02 15:50:33 +0000
+++ config.yaml	2016-01-09 14:43:24 +0000
@@ -122,3 +122,37 @@
       order for this charm to function correctly, the privacy extension must be
       disabled and a non-temporary address must be configured/available on
       your network interface.
+  # HA configuration settings
+  vip:
+    type: string
+    default:
+    description: |
+      Virtual IP(s) to use to front API services in HA configuration.
+      .
+      If multiple networks are being used, a VIP should be provided for each
+      network, separated by spaces.
+  vip_iface:
+    type: string
+    default: eth0
+    description: |
+      Default network interface to use for HA vip when it cannot be automatically
+      determined.
+  vip_cidr:
+    type: int
+    default: 24
+    description: |
+      Default CIDR netmask to use for HA vip when it cannot be automatically
+      determined.
+  ha-bindiface:
+    type: string
+    default: eth0
+    description: |
+      Default network interface on which HA cluster will bind to communication
+      with the other members of the HA Cluster.
+  ha-mcastport:
+    type: int
+    default: 5959
+    description: |
+      Default multicast port number that will be used to communicate between
+      HA Cluster nodes.
+

=== added symlink 'hooks/cluster-relation-changed'
=== target is u'heat_relations.py'
=== added symlink 'hooks/cluster-relation-departed'
=== target is u'heat_relations.py'
=== added symlink 'hooks/cluster-relation-joined'
=== target is u'heat_relations.py'
=== added symlink 'hooks/ha-relation-changed'
=== target is u'heat_relations.py'
=== added symlink 'hooks/ha-relation-joined'
=== target is u'heat_relations.py'
=== modified file 'hooks/heat_relations.py'
--- hooks/heat_relations.py	2015-10-30 11:22:56 +0000
+++ hooks/heat_relations.py	2016-01-09 14:43:24 +0000
@@ -19,7 +19,9 @@
     charm_dir,
     log,
     relation_ids,
+    relation_get,
     relation_set,
+    local_unit,
     open_port,
     unit_get,
     status_set,
@@ -35,6 +37,19 @@
     apt_update
 )
 
+from charmhelpers.contrib.hahelpers.cluster import (
+    is_elected_leader,
+    get_hacluster_config,
+)
+
+from charmhelpers.contrib.network.ip import (
+    get_iface_for_address,
+    get_netmask_for_address,
+    get_address_in_network,
+    get_ipv6_addr,
+    is_ipv6
+)
+
 from charmhelpers.contrib.openstack.utils import (
     configure_installation_source,
     openstack_upgrade_available,
@@ -55,6 +70,7 @@
     determine_packages,
     migrate_database,
     register_configs,
+    CLUSTER_RES,
     HEAT_CONF,
     REQUIRED_INTERFACES,
     setup_ipv6,
@@ -64,6 +80,7 @@
     API_PORTS,
 )
 
+from charmhelpers.contrib.openstack.context import ADDRESS_TYPES
 from charmhelpers.payload.execd import execd_preinstall
 
 hooks = Hooks()
@@ -108,6 +125,11 @@
     CONFIGS.write_all()
     configure_https()
 
+    for rid in relation_ids('cluster'):
+        cluster_joined(relation_id=rid)
+    for r_id in relation_ids('ha'):
+        ha_joined(relation_id=r_id)
+
 
 @hooks.hook('amqp-relation-joined')
 def amqp_joined(relation_id=None):
@@ -131,9 +153,9 @@
                                           config('database-user'),
                                           relation_prefix='heat')
     else:
-        relation_set(heat_database=config('database'),
-                     heat_username=config('database-user'),
-                     heat_hostname=unit_get('private-address'))
+        relation_set(database=config('database'),
+                     username=config('database-user'),
+                     hostname=unit_get('private-address'))
 
 
 @hooks.hook('shared-db-relation-changed')
@@ -143,7 +165,15 @@
         log('shared-db relation incomplete. Peer not ready?')
         return
     CONFIGS.write(HEAT_CONF)
-    migrate_database()
+
+    if is_elected_leader(CLUSTER_RES):
+        allowed_units = relation_get('allowed_units')
+        if allowed_units and local_unit() in allowed_units.split():
+            log('Cluster leader, performing db sync')
+            migrate_database()
+        else:
+            log('allowed_units either not presented, or local unit '
+                'not in acl list: %s' % repr(allowed_units))
 
 
 def configure_https():
@@ -216,6 +246,100 @@
     CONFIGS.write_all()
 
 
[e-mail]('cluster-relation-joined')
+def cluster_joined(relation_id=None):
+    for addr_type in ADDRESS_TYPES:
+        address = get_address_in_network(
+            config('os-{}-network'.format(addr_type))
+        )
+        if address:
+            relation_set(
+                relation_id=relation_id,
+                relation_settings={'{}-address'.format(addr_type): address}
+            )
+
+    if config('prefer-ipv6'):
+        private_addr = get_ipv6_addr(exc_list=[config('vip')])[0]
+        relation_set(relation_id=relation_id,
+                     relation_settings={'private-address': private_addr})
+
+
[e-mail]('cluster-relation-changed',
+            'cluster-relation-departed')
+@restart_on_change(restart_map(), stopstart=True)
+def cluster_changed():
+    CONFIGS.write_all()
+
+
[e-mail]('ha-relation-joined')
+def ha_joined(relation_id=None):
+    cluster_config = get_hacluster_config()
+
+    resources = {
+        'res_heat_haproxy': 'lsb:haproxy'
+    }
+
+    resource_params = {
+        'res_heat_haproxy': 'op monitor interval="5s"'
+    }
+
+    vip_group = []
+    for vip in cluster_config['vip'].split():
+        if is_ipv6(vip):
+            res_heat_vip = 'ocf:heartbeat:IPv6addr'
+            vip_params = 'ipv6addr'
+        else:
+            res_heat_vip = 'ocf:heartbeat:IPaddr2'
+            vip_params = 'ip'
+
+        iface = (get_iface_for_address(vip) or
+                 config('vip_iface'))
+        netmask = (get_netmask_for_address(vip) or
+                   config('vip_cidr'))
+
+        if iface is not None:
+            vip_key = 'res_heat_{}_vip'.format(iface)
+            resources[vip_key] = res_heat_vip
+            resource_params[vip_key] = (
+                'params {ip}="{vip}" cidr_netmask="{netmask}"'
+                ' nic="{iface}"'.format(ip=vip_params,
+                                        vip=vip,
+                                        iface=iface,
+                                        netmask=netmask)
+            )
+            vip_group.append(vip_key)
+
+    if len(vip_group) >= 1:
+        relation_set(relation_id=relation_id,
+                     groups={'grp_heat_vips': ' '.join(vip_group)})
+
+    init_services = {
+        'res_heat_haproxy': 'haproxy'
+    }
+    clones = {
+        'cl_heat_haproxy': 'res_heat_haproxy'
+    }
+    relation_set(relation_id=relation_id,
+                 init_services=init_services,
+                 corosync_bindiface=cluster_config['ha-bindiface'],
+                 corosync_mcastport=cluster_config['ha-mcastport'],
+                 resources=resources,
+                 resource_params=resource_params,
+                 clones=clones)
+
+
[e-mail]('ha-relation-changed')
+def ha_changed():
+    clustered = relation_get('clustered')
+    if not clustered or clustered in [None, 'None', '']:
+        log('ha_changed: hacluster subordinate not fully clustered.')
+    else:
+        log('Cluster configured, notifying other services and updating '
+            'keystone endpoint configuration')
+        for rid in relation_ids('identity-service'):
+            identity_joined(rid=rid)
+
+
 def main():
     try:
         hooks.execute(sys.argv)

=== modified file 'hooks/heat_utils.py'
--- hooks/heat_utils.py	2015-12-02 15:50:33 +0000
+++ hooks/heat_utils.py	2016-01-09 14:43:24 +0000
@@ -68,6 +68,8 @@
     'heat-engine'
 ]
 
+# Cluster resource used to determine leadership when hacluster'd
+CLUSTER_RES = 'grp_heat_vips'
 SVC = 'heat'
 HEAT_DIR = '/etc/heat'
 HEAT_CONF = '/etc/heat/heat.conf'
@@ -81,8 +83,7 @@
     (HEAT_CONF, {
         'services': BASE_SERVICES,
         'contexts': [context.AMQPContext(ssl_dir=HEAT_DIR),
-                     context.SharedDBContext(relation_prefix='heat',
-                                             ssl_dir=HEAT_DIR),
+                     context.SharedDBContext(ssl_dir=HEAT_DIR),
                      context.OSConfigFlagContext(),
                      HeatIdentityServiceContext(service=SVC, service_user=SVC),
                      HeatHAProxyContext(),

=== modified file 'metadata.yaml'
--- metadata.yaml	2015-11-18 10:35:35 +0000
+++ metadata.yaml	2016-01-09 14:43:24 +0000
@@ -14,3 +14,9 @@
     interface: rabbitmq
   identity-service:
     interface: keystone
+  ha:
+    interface: hacluster
+    scope: container
+peers:
+  cluster:
+    interface: heat-ha

=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-06-12 13:50:03 +0000
+++ tests/basic_deployment.py	2016-01-09 14:43:24 +0000
@@ -342,9 +342,9 @@
         relation = ['shared-db', 'mysql:shared-db']
         expected = {
             'private-address': u.valid_ip,
-            'heat_database': 'heat',
-            'heat_username': 'heat',
-            'heat_hostname': u.valid_ip
+            'database': 'heat',
+            'username': 'heat',
+            'hostname': u.valid_ip
         }
 
         ret = u.validate_relation_data(unit, relation, expected)
@@ -360,8 +360,8 @@
         expected = {
             'private-address': u.valid_ip,
             'db_host': u.valid_ip,
-            'heat_allowed_units': 'heat/0',
-            'heat_password': u.not_null
+            'allowed_units': 'heat/0',
+            'password': u.not_null
         }
 
         ret = u.validate_relation_data(unit, relation, expected)
@@ -465,7 +465,7 @@
         u.log.debug('mysql:heat relation: {}'.format(mysql_rel))
 
         db_uri = "mysql://{}:{}@{}/{}".format('heat',
-                                              mysql_rel['heat_password'],
+                                              mysql_rel['password'],
                                               mysql_rel['db_host'],
                                               'heat')
 

=== modified file 'unit_tests/test_heat_relations.py'
--- unit_tests/test_heat_relations.py	2015-11-16 09:14:33 +0000
+++ unit_tests/test_heat_relations.py	2016-01-09 14:43:24 +0000
@@ -41,6 +41,13 @@
     'execd_preinstall',
     'log',
     'migrate_database',
+    'is_elected_leader',
+    'relation_ids',
+    'relation_get',
+    'local_unit',
+    'get_hacluster_config',
+    'get_iface_for_address',
+    'get_netmask_for_address',
 ]
 
 
@@ -91,12 +98,14 @@
     def test_db_joined(self):
         self.unit_get.return_value = 'heat.foohost.com'
         relations.db_joined()
-        self.relation_set.assert_called_with(heat_database='heat',
-                                             heat_username='heat',
-                                             heat_hostname='heat.foohost.com')
+        self.relation_set.assert_called_with(database='heat',
+                                             username='heat',
+                                             hostname='heat.foohost.com')
         self.unit_get.assert_called_with('private-address')
 
     def _shared_db_test(self, configs):
+        self.relation_get.return_value = 'heat/0 heat/1'
+        self.local_unit.return_value = 'heat/0'
         configs.complete_contexts = MagicMock()
         configs.complete_contexts.return_value = ['shared-db']
         configs.write = MagicMock()
@@ -240,5 +249,44 @@
         self.sync_db_with_multi_ipv6_addresses.return_value = MagicMock()
         self.test_config.set('prefer-ipv6', True)
         relations.db_joined()
-        self.sync_db_with_multi_ipv6_addresses.assert_called_with_once(
-            'heat', 'heat')
+        self.sync_db_with_multi_ipv6_addresses.assert_called_with(
+            'heat', 'heat', relation_prefix='heat')
+
+    @patch.object(relations, 'CONFIGS')
+    def test_non_leader_db_changed(self, configs):
+        self.is_elected_leader.return_value = False
+        configs.complete_contexts.return_value = []
+        self.relation_get.return_value = 'heat/0 heat/1'
+        self.local_unit.return_value = 'heat/0'
+        configs.complete_contexts = MagicMock()
+        configs.complete_contexts.return_value = ['shared-db']
+        configs.write = MagicMock()
+        relations.db_changed()
+        self.assertFalse(self.migrate_database.called)
+
+    @patch.object(relations, 'CONFIGS')
+    def test_ha_joined(self, configs):
+        self.get_hacluster_config.return_value = {
+            'ha-bindiface': 'eth0',
+            'ha-mcastport': '5959',
+            'vip': '10.5.105.3'
+        }
+        self.get_iface_for_address.return_value = 'eth0'
+        self.get_netmask_for_address.return_value = '255.255.255.0'
+        relations.ha_joined()
+        expected = {
+            'relation_id': None,
+            'init_services': {'res_heat_haproxy': 'haproxy'},
+            'corosync_bindiface': 'eth0',
+            'corosync_mcastport': '5959',
+            'resources': {
+                'res_heat_haproxy': 'lsb:haproxy',
+                'res_heat_eth0_vip': 'ocf:heartbeat:IPaddr2'},
+            'resource_params': {
+                'res_heat_haproxy': 'op monitor interval="5s"',
+                'res_heat_eth0_vip': ('params ip="10.5.105.3" '
+                                      'cidr_netmask="255.255.255.0" '
+                                      'nic="eth0"')},
+            'clones': {'cl_heat_haproxy': 'res_heat_haproxy'}
+        }
+        self.relation_set.assert_called_with(**expected)