Merge lp:~gandelman-a/charms/precise/nova-cloud-controller/https_endpoint into lp:~openstack-charmers/charms/precise/nova-cloud-controller/ha-support

[Adam Gandelman]

nova-c-c HTTPS API support. These changes (and also those applied to the other chnages in similar merge proposals) allows the charm to dynamically manage the API listening port for each server depending on the current deployment. The canonical port in the Keystone catalog will always be the default (eg 8443) but the request pipeline on the API server differs depending on how it is deployed:

- When deployed in a single, non-https unit, the API server(s) will listen on their defualt port (eg, 8773)
- When new peers join, haproxy will be configured on every node to listen on the default port, with the local API server listening on $DEFAULT-10 (8773 -> 8763).
- When HTTPS is enabled and peers have joined, Apache is configured to terminate SSL on the default port and route requests to the local haproxy, then to the API server (8773 apache -> 8763 haproxy -> 8752 nova api).
- When HTTPS is enabled but no peers have joined, Apache terminates SSL on default port and routes requeusts directly to the API server (8773 -> 8763)

The same approach is used in the other HTTPS charms.

SSL certs are created and signed on the keystone side (when it is configured to do so) along with standard endpoint creation. An SSL cert, key and CA cert are returned to the unit along with its service credentials. Currently, nova-c-c will make a copy of the CA cert accessible to outside users at its web root.

This also adds a lot of general HA support that was committed to other charms but not to the nova-c-c (eligible_leader, etc).

[James Page (james-page)]

A few bits and pieces:

nova-cloud-controller-relations:ha_joined

Clones should be set to clones, not groups

775 - init_services="$init_services" groups="$groups"
776 + init_services="$init_services" clones="$groups"

nova-cloud-controller-relations:quantum_joined

I think the CA cert should be passed to quantum as well so it can talk back to the https server.

[James Page (james-page)]

I was able to get most of the https stuff working; two issues

1) Setting the config on keystone for https after building the environment configured endpoints in most places; however nova-cloud-controller did not reconfigure.

2) nova image-list post remove/add to fix the above resulted in:

2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack Traceback (most recent call last):
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return req.get_response(self.application)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack application, catch_exc_info=False)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return resp(environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 328, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return self.app(env, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return resp(environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return resp(environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return resp(environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return resp(environ, start_response)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-02-26 07:50:41.481 1056 TRACE nova.api.openstack return self.func(req, *ar...

[James Page (james-page)]

review: Needs Fixing

[lp:~gandelman-a/charms/precise/nova-cloud-controller/https_endpoint]

75. By Adam Gandelman on 2013-02-26

Actually send clone config to hacluster.

76. By Adam Gandelman on 2013-02-26

Send CA cert to quantum via quantum_joined, if it exists then.

77. By Adam Gandelman on 2013-02-26

Retrigger quantum_joined post-SSL.

78. By Adam Gandelman on 2013-02-26

Bump revision.

[Adam Gandelman (gandelman-a)]

I'm having a helluva time reproducing this issue. Can you describe how you initial built your environment before enabling HTTPS? Was nova-c-c peered and/or clustered? Are you sure you were deploying the most recent charm revision /w newest branch changes? bzr revs 71 and 72 dealt with an issue of the endpoint not reconfiguring after adding clustering, since the original reconfig hooks were not sending all required settings. Late-enabling HTTPS should be limited to identity-changed, though.

I've just tested again by enabling HTTPS after the entire catalog has been populated by haclustererd services (all endpoints pointing to http://$VIP/etc/). After nova-c-c has reconfigured its reverse proxying, it flushes the following settings back to KS, compute and quantum:

2013-02-27 15:09:31,482: hook.output@DEBUG: Flushed values for hook 'identity-service-relation-changed' on 'identity-service:27'
    Setting changed: 'ec2_admin_url'=u'https://192.168.77.2:8773/services/Cloud' (was 'http://192.168.77.2:8773/services/Cloud')
    Setting changed: 'ec2_internal_url'=u'https://192.168.77.2:8773/services/Cloud' (was 'http://192.168.77.2:8773/services/Cloud')
    Setting changed: 'ec2_public_url'=u'https://192.168.77.2:8773/services/Cloud' (was 'http://192.168.77.2:8773/services/Cloud')
    Setting changed: 'nova_admin_url'=u'https://192.168.77.2:8774/v1.1/$(tenant_id)s' (was 'http://192.168.77.2:8774/v1.1/$(tenant_id)s')
    Setting changed: 'nova_internal_url'=u'https://192.168.77.2:8774/v1.1/$(tenant_id)s' (was 'http://192.168.77.2:8774/v1.1/$(tenant_id)s')
    Setting changed: 'nova_public_url'=u'https://192.168.77.2:8774/v1.1/$(tenant_id)s' (was 'http://192.168.77.2:8774/v1.1/$(tenant_id)s')
    Setting changed: 'quantum_admin_url'=u'https://192.168.77.2:9696' (was 'http://192.168.77.2:9696')
    Setting changed: 'quantum_internal_url'=u'https://192.168.77.2:9696' (was 'http://192.168.77.2:9696')
    Setting changed: 'quantum_public_url'=u'https://192.168.77.2:9696' (was 'http://192.168.77.2:9696')
    Setting changed: 's3_admin_url'=u'https://192.168.77.2:3333' (was 'http://192.168.77.2:3333')
    Setting changed: 's3_internal_url'=u'https://192.168.77.2:3333' (was 'http://192.168.77.2:3333')
    Setting changed: 's3_public_url'=u'https://192.168.77.2:3333' (was 'http://192.168.77.2:3333')
    Setting changed: u'ca_cert'=u'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNvekNDQWd5Z0F3SUJBZ0lCQVRBTkJna3Fo\na2lHOXcwQkFRVUZBREJy (was unset) on 'cloud-compute:47'
    Setting changed: 'quantum_url'=u'https://192.168.77.2:9696' (was 'http://192.168.77.2:9696') on 'cloud-compute:47'
    Setting changed: u'ca_cert'=u'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNvekNDQWd5Z0F3SUJBZ0lCQVRBTkJna3Fo\na2lHOXcwQkFRVUZBREJy (was unset) on 'quantum-network-service:40'
    Setting changed: 'quantum_url'=u'https://192.168.77.2:9696' (was 'http://192.168.77.2:9696') on 'quantum-network-service:40'
2013-02-27 15:09:31,483: hook.executor@DEBUG: Hook complete: /var/lib/juju/units/nova-cloud-controller-
2/charm/hooks/identity-service-relation-changed

If you can test once again, and possibly keep an eye on the nova-c-c leader's charm log, to see what settings change? The r...

[lp:~gandelman-a/charms/precise/nova-cloud-controller/https_endpoint]

79. By Adam Gandelman on 2013-03-08

Rebase against current ha-support branch.

80. By Adam Gandelman on 2013-03-09

Block configure_qunatum_networking if no amqp relation exists.

81. By Adam Gandelman on 2013-03-09

Avoid race between https + quantum network setup.

using https() in keystone_joined() is not reliable with multiple
KS peers. Only inspect local config and count on https() only from
_changed() hooks.

82. By Adam Gandelman on 2013-03-09

Bump rev.

83. By Adam Gandelman on 2013-03-09

Only configurate quantum net. from keystone-changed when safe.

They ks catalog may be in a state of flux when these hooks fire.

If firing for the first time, configure quantum networking before
https frontend is setup, and before KS endpoint is reconfigured to
point to https.

If firing as the result of HA reconfiguration, configure quantum
after HTTPS has been updated to include updated certificate (for VIP
address).

84. By Adam Gandelman on 2013-03-09

keystone_changed: Protect late quantum config by an eligible_leader check.

85. By Adam Gandelman on 2013-03-09

Also protect early call to quantum config.

86. By Adam Gandelman on 2013-03-09

keystone_changed: Also determine HTTPS based on actual hook being fired.

87. By Adam Gandelman on 2013-03-09

Be careful of 'set -e'.

=== modified file 'config.yaml'
--- config.yaml	2013-01-22 16:43:49 +0000
+++ config.yaml	2013-03-09 04:30:28 +0000
@@ -128,3 +128,13 @@
     description: |
        Default multicast port number that will be used to communicate between
        HA Cluster nodes.
+  ssl_cert:
+    type: string
+    description: |
+      SSL certificate to install and use for API ports.  Setting this value
+      and ssl_key will enable reverse proxying, point Glance's entry in the
+      Keystone catalog to use https, and override any certficiate and key
+      issued by Keystone (if it is configured to do so).
+  ssl_key:
+    type: string
+    description: SSL key to use with certificate specified as ssl_cert.

=== added symlink 'hooks/cluster-relation-changed'
=== target is u'nova-cloud-controller-relations'
=== added symlink 'hooks/cluster-relation-departed'
=== target is u'nova-cloud-controller-relations'
=== added symlink 'hooks/ha-relation-changed'
=== target is u'nova-cloud-controller-relations'
=== added symlink 'hooks/ha-relation-joined'
=== target is u'nova-cloud-controller-relations'
=== modified file 'hooks/lib/openstack-common'
--- hooks/lib/openstack-common	2013-03-08 21:18:29 +0000
+++ hooks/lib/openstack-common	2013-03-09 04:30:28 +0000
@@ -321,7 +321,6 @@
 
 HAPROXY_CFG=/etc/haproxy/haproxy.cfg
 HAPROXY_DEFAULT=/etc/default/haproxy
-
 ##########################################################################
 # Description: Configures HAProxy services for Openstack API's
 # Parameters:
@@ -330,9 +329,8 @@
 #   assumes the name of the peer relation is 'cluster' and that every
 #   service unit in the peer relation is running the same services.
 #
-#   The HAProxy service will listen on port + 10000.
-# Example:
-#   configure_haproxy cinder_api:12345 nova_api:9999
+# Example
+#   configure_haproxy cinder_api:8776:8756i nova_api:8774:8764
 ##########################################################################
 configure_haproxy() {
   local address=`unit-get private-address`
@@ -368,14 +366,18 @@
 EOF
   for service in $@; do
     local service_name=$(echo $service | cut -d : -f 1)
-    local api_listen_port=$(echo $service | cut -d : -f 2)
-    local haproxy_listen_port=$(($api_listen_port + 10000))
+    local haproxy_listen_port=$(echo $service | cut -d : -f 2)
+    local api_listen_port=$(echo $service | cut -d : -f 3)
+    juju-log "Adding haproxy configuration entry for $service "\
+             "($haproxy_listen_port -> $api_listen_port)"
     cat >> $HAPROXY_CFG << EOF
 listen $service_name 0.0.0.0:$haproxy_listen_port
   balance roundrobin
   option tcplog
   server $name $address:$api_listen_port check
 EOF
+    local r_id=""
+    local unit=""
     for r_id in `relation-ids cluster`; do
       for unit in `relation-list -r $r_id`; do
         local unit_name=${unit////-}
@@ -388,6 +390,7 @@
     done
   done
   echo "ENABLED=1" > $HAPROXY_DEFAULT
+  service haproxy restart
 }
 
 ##########################################################################
@@ -395,18 +398,20 @@
 # Returns: 0 if configured, 1 if not configured
 ##########################################################################
 is_clustered() {
+  local r_id=""
+  local unit=""
   for r_id in $(relation-ids ha); do
     if [ -n "$r_id" ]; then
       for unit in $(relation-list -r $r_id); do
          clustered=$(relation-get -r $r_id clustered $unit)
          if [ -n "$clustered" ]; then
-           echo "Unit is clustered"
+           juju-log "Unit is haclustered"
            return 0
          fi
       done
     fi
   done
-  echo "Unit is not clustered"
+  juju-log "Unit is not haclustered"
   return 1
 }
 
@@ -415,6 +420,7 @@
 ##########################################################################
 peer_units() {
   local peers=""
+  local r_id=""
   for r_id in $(relation-ids cluster); do
     peers="$peers $(relation-list -r $r_id)"
   done
@@ -433,11 +439,11 @@
     echo "Comparing $JUJU_UNIT_NAME with peers: $peers"
     local r_unit_no=$(echo $peer | cut -d / -f 2)
     if (($r_unit_no<$l_unit_no)); then
-        echo "Not oldest peer; deferring"
+        juju-log "Not oldest peer; deferring"
         return 1
     fi
   done
-  echo "Oldest peer; might take charge?"
+  juju-log "Oldest peer; might take charge?"
   return 0
 }
 
@@ -451,13 +457,13 @@
 eligible_leader() {
   if is_clustered; then
     if ! is_leader $1; then
-      echo 'Deferring action to CRM leader'
+      juju-log 'Deferring action to CRM leader'
       return 1
     fi
   else
     peers=$(peer_units)
     if [ -n "$peers" ] && ! oldest_peer "$peers"; then
-      echo 'Deferring action to oldest service unit.'
+      juju-log 'Deferring action to oldest service unit.'
       return 1
     fi
   fi
@@ -469,14 +475,14 @@
 # Returns: 0 if peered, 1 if not peered
 ##########################################################################
 is_peered() {
-  r_id=$(relation-ids cluster)
+  local r_id=$(relation-ids cluster)
   if [ -n "$r_id" ]; then
     if [ -n "$(relation-list -r $r_id)" ]; then
-      echo "Unit peered"
+      juju-log "Unit peered"
       return 0
     fi
   fi
-  echo "Unit not peered"
+  juju-log "Unit not peered"
   return 1
 }
 
@@ -489,12 +495,207 @@
   hostname=`hostname`
   if [ -x /usr/sbin/crm ]; then
     if crm resource show $1 | grep -q $hostname; then
-      echo "$hostname is cluster leader"
+      juju-log "$hostname is cluster leader."
       return 0
     fi
   fi
-  echo "$hostname is not cluster leader"
-  return 1
+  juju-log "$hostname is not cluster leader."
+  return 1
+}
+
+##########################################################################
+# Description: Determines whether enough data has been provided in
+# configuration or relation data to configure HTTPS.
+# Parameters:  None
+# Returns: 0 if HTTPS can be configured, 1 if not.
+##########################################################################
+https() {
+  local r_id=""
+  if [[ -n "$(config-get ssl_cert)" ]] &&
+     [[ -n "$(config-get ssl_key)" ]] ; then
+    return 0
+  fi
+  for r_id in $(relation-ids identity-service) ; do
+    for unit in $(relation-list -r $r_id) ; do
+      if [[ "$(relation-get -r $r_id https_keystone $unit)" == "True" ]] &&
+         [[ -n "$(relation-get -r $r_id ssl_cert $unit)" ]] &&
+         [[ -n "$(relation-get -r $r_id ssl_key $unit)" ]] &&
+         [[ -n "$(relation-get -r $r_id ca_cert $unit)" ]] ; then
+          return 0
+      fi
+    done
+  done
+  return 1
+}
+
+##########################################################################
+# Description: For a given number of port mappings, configures apache2
+# HTTPs local reverse proxying using certficates and keys provided in
+# either configuration data (preferred) or relation data.  Assumes ports
+# are not in use (calling charm should ensure that).
+# Parameters:  Variable number of proxy port mappings as
+# $internal:$external.
+# Returns: 0 if reverse proxy(s) have been configured, 0 if not.
+##########################################################################
+enable_https() {
+  local port_maps="$@"
+  local http_restart=""
+  juju-log "Enabling HTTPS for port mappings: $port_maps."
+
+  # allow overriding of keystone provided certs with those set manually
+  # in config.
+  local cert=$(config-get ssl_cert)
+  local key=$(config-get ssl_key)
+  local ca_cert=""
+  if [[ -z "$cert" ]] || [[ -z "$key" ]] ; then
+    juju-log "Inspecting identity-service relations for SSL certificate."
+    local r_id=""
+    cert=""
+    key=""
+    ca_cert=""
+    for r_id in $(relation-ids identity-service) ; do
+      for unit in $(relation-list -r $r_id) ; do
+        [[ -z "$cert" ]] && cert="$(relation-get -r $r_id ssl_cert $unit)"
+        [[ -z "$key" ]] && key="$(relation-get -r $r_id ssl_key $unit)"
+        [[ -z "$ca_cert" ]] && ca_cert="$(relation-get -r $r_id ca_cert $unit)"
+      done
+    done
+    [[ -n "$cert" ]] && cert=$(echo $cert | base64 -di)
+    [[ -n "$key" ]] && key=$(echo $key | base64 -di)
+    [[ -n "$ca_cert" ]] && ca_cert=$(echo $ca_cert | base64 -di)
+  else
+    juju-log "Using SSL certificate provided in service config."
+  fi
+
+  [[ -z "$cert" ]] || [[ -z "$key" ]] &&
+    juju-log "Expected but could not find SSL certificate data, not "\
+             "configuring HTTPS!" && return 1
+
+  apt-get -y install apache2
+  a2enmod ssl proxy proxy_http | grep -v "To activate the new configuration" &&
+    http_restart=1
+
+  mkdir -p /etc/apache2/ssl/$CHARM
+  echo "$cert" >/etc/apache2/ssl/$CHARM/cert
+  echo "$key" >/etc/apache2/ssl/$CHARM/key
+  if [[ -n "$ca_cert" ]] ; then
+    juju-log "Installing Keystone supplied CA cert."
+    echo "$ca_cert" >/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt
+    update-ca-certificates --fresh
+
+    # XXX TODO: Find a better way of exporting this?
+    if [[ "$CHARM" == "nova-cloud-controller" ]] ; then
+      [[ -e /var/www/keystone_juju_ca_cert.crt ]] &&
+        rm -rf /var/www/keystone_juju_ca_cert.crt
+      ln -s  /usr/local/share/ca-certificates/keystone_juju_ca_cert.crt \
+             /var/www/keystone_juju_ca_cert.crt
+    fi
+
+  fi
+  for port_map in $port_maps ; do
+    local ext_port=$(echo $port_map | cut -d: -f1)
+    local int_port=$(echo $port_map | cut -d: -f2)
+    juju-log "Creating apache2 reverse proxy vhost for $port_map."
+    cat >/etc/apache2/sites-available/${CHARM}_${ext_port} <<END
+Listen $ext_port
+NameVirtualHost *:$ext_port
+<VirtualHost *:$ext_port>
+    ServerName $(unit-get private-address)
+    SSLEngine on
+    SSLCertificateFile /etc/apache2/ssl/$CHARM/cert
+    SSLCertificateKeyFile /etc/apache2/ssl/$CHARM/key
+    ProxyPass / http://localhost:$int_port/
+    ProxyPassReverse / http://localhost:$int_port/
+    ProxyPreserveHost on
+</VirtualHost>
+<Proxy *>
+    Order deny,allow
+    Allow from all
+</Proxy>
+<Location />
+    Order allow,deny
+    Allow from all
+</Location>
+END
+    a2ensite ${CHARM}_${ext_port} | grep -v "To activate the new configuration" &&
+      http_restart=1
+  done
+  if [[ -n "$http_restart" ]] ; then
+    service apache2 restart
+  fi
+}
+
+##########################################################################
+# Description: Ensure HTTPS reverse proxying is disabled for given port
+# mappings.
+# Parameters:  Variable number of proxy port mappings as
+# $internal:$external.
+# Returns: 0 if reverse proxy is not active for all portmaps, 1 on error.
+##########################################################################
+disable_https() {
+  local port_maps="$@"
+  local http_restart=""
+  juju-log "Ensuring HTTPS disabled for $port_maps."
+  ( [[ ! -d /etc/apache2 ]] || [[ ! -d /etc/apache2/ssl/$CHARM ]] ) && return 0
+  for port_map in $port_maps ; do
+    local ext_port=$(echo $port_map | cut -d: -f1)
+    local int_port=$(echo $port_map | cut -d: -f2)
+    if [[ -e /etc/apache2/sites-available/${CHARM}_${ext_port} ]] ; then
+      juju-log "Disabling HTTPS reverse proxy for $CHARM $port_map."
+      a2dissite ${CHARM}_${ext_port} | grep -v "To activate the new configuration" &&
+        http_restart=1
+    fi
+  done
+  if [[ -n "$http_restart" ]] ; then
+    service apache2 restart
+  fi
+}
+
+
+##########################################################################
+# Description: Ensures HTTPS is either enabled or disabled for given port
+# mapping.
+# Parameters:  Variable number of proxy port mappings as
+# $internal:$external.
+# Returns: 0 if HTTPS reverse proxy is in place, 1 if it is not.
+##########################################################################
+setup_https() {
+  # configure https via apache reverse proxying either
+  # using certs provided by config or keystone.
+  [[ -z "$CHARM" ]] &&
+    error_out "setup_https(): CHARM not set."
+  if ! https ; then
+    disable_https $@
+  else
+    enable_https $@
+  fi
+}
+
+##########################################################################
+# Description: Determine correct API server listening port based on
+# existence of HTTPS reverse proxy and/or haproxy.
+# Paremeters: The standard public port for given service.
+# Returns: The correct listening port for API service.
+##########################################################################
+determine_api_port() {
+  local public_port="$1"
+  local i=0
+  ( [[ -n "$(peer_units)" ]] || is_clustered >/dev/null 2>&1 ) && i=$[$i + 1]
+  https >/dev/null 2>&1 && i=$[$i + 1]
+  echo $[$public_port - $[$i * 10]]
+}
+
+##########################################################################
+# Description: Determine correct proxy listening port based on public IP +
+# existence of HTTPS reverse proxy.
+# Paremeters: The standard public port for given service.
+# Returns: The correct listening port for haproxy service public address.
+##########################################################################
+determine_haproxy_port() {
+  local public_port="$1"
+  local i=0
+  https >/dev/null 2>&1 && i=$[$i + 1]
+  echo $[$public_port - $[$i * 10]]
 }
 
 ##########################################################################

=== modified file 'hooks/nova-cloud-controller-common'
--- hooks/nova-cloud-controller-common	2013-01-18 12:22:44 +0000
+++ hooks/nova-cloud-controller-common	2013-03-09 04:30:28 +0000
@@ -97,6 +97,7 @@
   if [ "$(config-get conf-ext-net)" != "no" ] &&
      [ "$QUANTUM_PLUGIN" == "ovs" ] &&
      [ -f /etc/quantum/novarc ] &&
+     [ -n "$(relation-ids amqp)" ] &&
      [ -n "$(relation-ids shared-db)" ]; then
     juju-log "Configuring external networking for quantum"
     # Use helper to create external network gateway
@@ -219,3 +220,73 @@
     known_hosts="$(base64 /etc/nova/compute_ssh/$sunit/known_hosts)" \
     authorized_keys="$(base64 /etc/nova/compute_ssh/$sunit/authorized_keys)"
 }
+
+configure_https() {
+  # setup https termination for all api services, depending on what is running
+  # and topology of current deployment.
+  local clustered=""
+  ( [[ -n "$(peer_units)" ]] || is_clustered ) && clustered="1"
+  local services=""
+  local ssl_port_maps=""
+  local haproxy_port_maps=""
+  local next_server=""
+  local api_port=""
+
+  # upstartService:defaultPort:configOption
+  local svcs="nova-api-ec2:8773:ec2_listen_port
+              nova-api-os-compute:8774:osapi_compute_listen_port
+              nova-objectstore:3333:s3_listen_port"
+  [[ "$NET_MANAGER" == "Quantum" ]] &&
+    svcs="$svcs quantum-server:9696:bind_port"
+
+  for s in $svcs ; do
+    local service=$(echo $s | cut -d: -f1)
+    local port=$(echo $s | cut -d: -f2)
+    local opt=$(echo $s | cut -d: -f3)
+    if [[ -n "$clustered" ]] ; then
+      next_server="$(determine_haproxy_port $port)"
+      api_port="$(determine_api_port $port)"
+      haproxy_port_maps="$haproxy_port_maps $service:$next_server:$api_port"
+    else
+      api_port="$(determine_api_port $port)"
+      next_server="$api_port"
+    fi
+    if [[ "$service" == "quantum-server" ]] ; then
+      set_or_update "$opt" "$api_port" "$QUANTUM_CONF"
+    else
+      set_or_update "$opt" "$api_port"
+    fi
+    ssl_port_maps="$ssl_port_maps $port:$next_server"
+  done
+
+  # make sure all backend api servers are bound to new backend port
+  # before setting up any frontends.
+  for s in $svcs ; do
+    local service=$(echo $s | cut -d: -f1)
+    service_ctl $service restart
+  done
+
+  [[ -n "$haproxy_port_maps" ]] && configure_haproxy $haproxy_port_maps
+  setup_https $ssl_port_maps
+
+  # another restart to ensure api servers are now bound to frontend ports
+  # that may have just been disabled.
+  for s in $svcs ; do
+    local service=$(echo $s | cut -d: -f1)
+    service_ctl $service restart
+  done
+
+  local r_id=""
+  # (re)configure ks endpoint accordingly
+  for r_id in $(relation-ids identity-service) ; do
+    keystone_joined "$r_id"
+  done
+  # pass on possibly updated quantum URL + ca_cert to compute nodes.
+  for r_id in $(relation-ids cloud-compute) ; do
+    compute_joined "$r_id"
+  done
+  # update the quantum relation, as well.
+  for r_id in $(relation-ids quantum-network-service) ; do
+    quantum_joined "$r_id"
+  done
+}

=== modified file 'hooks/nova-cloud-controller-relations'
--- hooks/nova-cloud-controller-relations	2013-02-21 23:12:04 +0000
+++ hooks/nova-cloud-controller-relations	2013-03-09 04:30:28 +0000
@@ -44,6 +44,7 @@
   cp files/create_tenant_net.py /usr/bin/quantum-tenant-net
 
   service_ctl all stop
+  configure_https
 }
 
 function upgrade_charm {
@@ -70,13 +71,14 @@
   set_config_flags
 
   if [ "$NET_MANAGER" == "Quantum" ] && \
-     is_clustered && is_leader 'res_nova_vip' || \
+     eligible_leader 'res_nova_vip' || \
      ! is_clustered; then
     configure_quantum_networking
   fi
 
   determine_services
   service_ctl all restart
+  configure_https
 }
 
 function amqp_joined {
@@ -131,6 +133,12 @@
   fi
 
   determine_services && service_ctl all restart
+
+  if [ "$NET_MANAGER" == "Quantum" ] && \
+     eligible_leader 'res_nova_vip' || \
+     ! is_clustered; then
+    configure_quantum_networking
+  fi
 }
 
 function db_joined {
@@ -170,11 +178,17 @@
   fi
   determine_services
   service_ctl all stop
-  /usr/bin/nova-manage db sync
+
+  eligible_leader 'res_nova_vip' && /usr/bin/nova-manage db sync
+
   service_ctl all start
-  if [ "$NET_MANAGER" == "Quantum" ]; then
+
+  if [ "$NET_MANAGER" == "Quantum" ] && \
+     eligible_leader 'res_nova_vip' || \
+     ! is_clustered; then
     configure_quantum_networking
   fi
+
   trigger_remote_service_restarts
 }
 
@@ -191,28 +205,37 @@
   # we need to get two entries into keystone's catalog, nova + ec2
   # group, them by prepending $service_ to each setting. the keystone
   # charm will assemble settings into corresponding catalog entries
-  if is_clustered && is_leader 'res_nova_vip'; then
-    address=$(config-get vip)
-    nova_port=18774
-    ec2_port=18773
-    s3_port=13333
-    quantum_port=19696
-    vol_port=18776
-  elif ! is_clustered; then
-    address=$(unit-get private-address)
-    nova_port=8774
-    ec2_port=8773
-    s3_port=3333
-    quantum_port=9696
-    vol_port=8776
+  eligible_leader 'res_nova_vip' || return 0
+
+  is_clustered && local host=$(config-get vip) ||
+    local host=$(unit-get private-address)
+
+  if [[ "$arg0" == "identity-service-relation-joined" ]] ; then
+    # determine https status based only on config at this point,
+    # insepcting KS relation is not reliable.  if KS has mulitple
+    # units, multiple relation-joineds are fired, resulting in the
+    # endpoint being configured in catalog as https before https
+    # is actually setup on this end. ends with failure to configure
+    # quantum network, if its enabled.
+    # if specified in config, https will have already been setup in
+    # install or config-changed.
+    if [[ -n "$(config-get ssl_cert)" ]] &&
+       [[ -n "$(config-get ssl_key)" ]] ; then
+      local scheme="https"
+    else
+      local scheme="http"
+    fi
   else
-    # Not the leader and clustered - no action required
-    return 0
+    # this function is called from other hook contexts, use normal method
+    # for determining https
+    https && scheme="https" || scheme="http"
   fi
-  nova_url="http://$address:$nova_port/v1.1/\$(tenant_id)s"
-  ec2_url="http://$address:$ec2_port/services/Cloud"
-  s3_url="http://$address:$s3_port"
-  region="$(config-get region)"
+
+  local nova_url="$scheme://$host:8774/v1.1/\$(tenant_id)s"
+  local ec2_url="$scheme://$host:8773/services/Cloud"
+  local s3_url="$scheme://$host:3333"
+  local region="$(config-get region)"
+  local quantum_url="$scheme://$host:9696"
 
   # these are the default endpoints
   relation-set nova_service="nova" \
@@ -232,7 +255,6 @@
     s3_internal_url="$s3_url"
 
   if [ "$(config-get network-manager)" == "Quantum" ]; then
-    quantum_url="http://$address:$quantum_port"
     relation-set quantum_service="quantum" \
         quantum_region="$region" \
         quantum_public_url="$quantum_url" \
@@ -242,7 +264,7 @@
 
   # tack on an endpoint for nova-volume a relation exists.
   if [[ -n "$(relation-ids nova-volume-service)" ]] ; then
-    nova_vol_url="http://$address:$vol_port/v1/\$(tenant_id)s"
+    nova_vol_url="$scheme://$host:$vol_port/v1/\$(tenant_id)s"
     relation-set nova-volume_service="nova-volume" \
       nova-volume_region="$region" \
       nova-volume_public_url="$nova_vol_url" \
@@ -281,6 +303,13 @@
     sed -i '/--use_deprecated_auth/d' $NOVA_CONF
   fi
 
+  local clustered=""
+  is_clustered && clustered="1"
+
+  [[ -n "$clustered" ]]  && local host=$(config-get vip) ||
+    local host=$(unit-get private-address)
+  https && local scheme="https" || local scheme="http"
+
   # update keystone authtoken settings accordingly
   set_or_update "service_host" "$service_host" "$API_CONF"
   set_or_update "service_port" "$service_port" "$API_CONF"
@@ -296,7 +325,7 @@
   if [ "$NET_MANAGER" == "Quantum" ]; then
     # Configure Nova for quantum
     keystone_url="http://${auth_host}:${auth_port}/v2.0"
-    set_or_update "quantum_url" "http://$(unit-get private-address):9696"
+    set_or_update "quantum_url" "$scheme://$host:9696"
     set_or_update "quantum_admin_tenant_name" "${service_tenant}"
     set_or_update "quantum_admin_username" "${service_username}"
     set_or_update "quantum_admin_password" "${service_password}"
@@ -320,11 +349,15 @@
   determine_services && service_ctl all restart
 
   if [ "$NET_MANAGER" == "Quantum" ]; then
-    configure_quantum_networking
+    # if first time here, config quantum before setting up
+    # https.
+    if [[ -z "$clustered" ]] && eligible_leader ; then
+      configure_quantum_networking
+    fi
     # ripple out changes to identity to connected services
     # which use cloud-controller as source of information for
     # keystone
-    r_ids="$(relation-ids cloud-compute) $(relation-ids quantum-network-service)"
+    local r_ids="$(relation-ids cloud-compute) $(relation-ids quantum-network-service)"
     for id in $r_ids ; do
       relation-set -r $id \
         keystone_host=$auth_host \
@@ -339,6 +372,15 @@
 
     done
   fi
+  configure_https
+
+  # if this changed event happens as a result of clustered VIP
+  # reconfigure, configure_https needs to update VIP certificate
+  # before quantumclient is used.
+  if [[ "$NET_MANAGER" == "Quantum" ]] &&
+     [[ -n "$clustered" ]] && eligible_leader 'res_nova_vip' ; then
+      configure_quantum_networking
+  fi
 }
 
 volume_joined() {
@@ -390,19 +432,19 @@
 }
 
 compute_joined() {
-  if is_clustered && ! is_leader 'res_nova_vip'; then
-    # Clustered and not current leader - do nothing
-    return 0
-  fi
-
-  relation-set network_manager=$(config-get network-manager)
-  relation-set ec2_host=$(unit-get private-address)
+  local r_id="$1"
+  [[ -n "$r_id" ]] && r_id="-r $r_id"
+  eligible_leader 'res_nova_vip' || return 0
+  relation-set $r_id network_manager=$(config-get network-manager)
+  # XXX Should point to VIP if clustered, or this may not even be needed.
+  relation-set $r_id ec2_host=$(unit-get private-address)
 
   local sect="filter:authtoken"
   keystone_host=$(local_config_get $API_CONF auth_host $sect)
+
   if [ "$NET_MANAGER" == "Quantum" ]; then
-    if [ -n "$keystone_host" ]; then
-      relation-set \
+    if [[ -n "$keystone_host" ]]; then
+      relation-set $r_id \
         keystone_host=$keystone_host \
         auth_port=$(local_config_get $API_CONF auth_port $sect) \
         service_port=$(local_config_get $API_CONF service_port $sect) \
@@ -410,19 +452,24 @@
         service_password=$(local_config_get $API_CONF admin_password $sect) \
         service_tenant=$(local_config_get $API_CONF admin_tenant_name $sect) \
         auth_uri=$(local_config_get $API_CONF auth_uri $sect)
-    fi
-
-    if is_clustered; then
-      quantum_host=$(config-get vip)
-      quantum_port=19696
-    else
-      quantum_host=$(unit-get private-address)
-      quantum_port=9696
-    fi
-
-    relation-set quantum_host=$quantum_host \
-                 quantum_port=$quantum_port \
-                 quantum_plugin=$(config-get quantum-plugin)
+
+    fi
+    is_clustered && local host=$(config-get vip) ||
+      local host=$(unit-get private-address)
+    https && local scheme="https" || local scheme="http"
+    local quantum_url="$scheme://$host:9696"
+
+    relation-set $r_id quantum_url=$quantum_url \
+                       quantum_plugin=$(config-get quantum-plugin) \
+                       region=$(config-get region)
+
+  fi
+
+  # must pass on the keystone CA certficiate, if it exists.
+  cert="/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt"
+  if [[ -n "$keystone_host" ]] && [[ -e $cert ]] ; then
+    cert=$(cat $cert | base64)
+    relation-set $r_id ca_cert="$cert"
   fi
 
   # volume driver is dependent on os version, or presence
@@ -434,11 +481,10 @@
       vol_drv="nova-volume"
       ;;
     "folsom")
-      local r_ids=$(relation-ids cinder-volume-service)
-      [[ -z "$r_ids" ]] && vol_drv="nova-volume"
+      [[ -z "$(relation-ids cinder-volume-service)" ]] && vol_drv="nova-volume"
       ;;
   esac
-  relation-set volume_service="$vol_drv"
+  relation-set $r_id volume_service="$vol_drv"
 }
 
 compute_changed() {
@@ -458,15 +504,14 @@
 
 function quantum_joined() {
   # Tell quantum service about keystone
-  if is_clustered && ! is_leader 'res_nova_vip'; then
-    # Clustered and not current leader - do nothing
-    return 0
-  fi
+  eligible_leader || return 0
+  local r_id="$1"
+  [[ -n "$r_id" ]] && r_id="-r $r_id"
 
   local sect="filter:authtoken"
   keystone_host=$(local_config_get $API_CONF auth_host $sect)
   if [ -n "$keystone_host" ]; then
-    relation-set \
+    relation-set $r_id \
       keystone_host=$keystone_host \
       auth_port=$(local_config_get $API_CONF auth_port $sect) \
       service_port=$(local_config_get $API_CONF service_port $sect) \
@@ -476,24 +521,50 @@
       auth_uri=$(local_config_get $API_CONF auth_uri $sect)
   fi
 
-  if is_clustered; then
-    quantum_host=$(config-get vip)
-    quantum_port=19696
-  else
-    quantum_host=$(unit-get private-address)
-    quantum_port=9696
+  # must pass on the keystone CA certficiate, if it exists.
+  cert="/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt"
+  if [[ -n "$keystone_host" ]] && [[ -e $cert ]] ; then
+    cert=$(cat $cert | base64)
+    relation-set $r_id ca_cert="$cert"
   fi
 
-  relation-set quantum_host=$quantum_host \
-               quantum_port=$quantum_port \
-               quantum_plugin=$(config-get quantum-plugin) \
-               region=$(config-get region)
+  is_clustered && local host=$(config-get vip) ||
+    local host=$(unit-get private-address)
+  https && local scheme="https" || local scheme="http"
+  local quantum_url="$scheme://$host:9696"
+
+  relation-set $r_id quantum_url=$quantum_url \
+                     quantum_plugin=$(config-get quantum-plugin) \
+                     region=$(config-get region)
+
 }
 
 function cluster_changed() {
-  configure_haproxy "quantum_api:9696" "nova_api:8774" \
-                    "ec2_api:8773" "s3_api:3333" \
-                    "volume_api:8776"
+  [[ -z "$(peer_units)" ]] &&
+    juju-log "cluster_changed() with no peers." && exit 0
+  # upstartService:defaultPort:configOption
+  local svcs="nova-api-ec2:8773:ec2_listen_port
+              nova-api-os-compute:8774:osapi_compute_listen_port
+              nova-objectstore:3333:s3_listen_port"
+  [[ "$NET_MANAGER" == "Quantum" ]] &&
+    svcs="$svcs quantum-server:9696:bind_port"
+
+  for s in $svcs ; do
+    local service=$(echo $s | cut -d: -f1)
+    local port=$(echo $s | cut -d: -f2)
+    local opt=$(echo $s | cut -d: -f3)
+    local next_server="$(determine_haproxy_port $port)"
+    local api_port="$(determine_api_port $port)"
+    local haproxy_port_maps="$haproxy_port_maps $service:$next_server:$api_port"
+    if [[ "$service" == "quantum-server" ]] ; then
+      set_or_update "$opt" "$api_port" "$QUANTUM_CONF"
+    else
+      set_or_update "$opt" "$api_port"
+    fi
+
+    service_ctl $service restart
+  done
+  configure_haproxy $haproxy_port_maps
 }
 
 function ha_relation_joined() {
@@ -518,13 +589,13 @@
     init_services="{
 'res_nova_haproxy':'haproxy'
 }"
-    groups="{
-'grp_nova_haproxy':'res_nova_vip res_nova_haproxy'
+    clones="{
+'cl_nova_haproxy':'res_nova_haproxy'
 }"
     relation-set corosync_bindiface=$corosync_bindiface \
       corosync_mcastport=$corosync_mcastport \
       resources="$resources" resource_params="$resource_params" \
-      init_services="$init_services" groups="$groups"
+      init_services="$init_services" clones="$clones"
   else
     juju-log "Insufficient configuration data to configure hacluster"
     exit 1
@@ -534,45 +605,57 @@
 function ha_relation_changed() {
   local clustered=`relation-get clustered`
   if [ -n "$clustered" ] && is_leader 'res_nova_vip'; then
+    https && local scheme="https" || local scheme="http"
     for r_id in `relation-ids identity-service`; do
-      address=$(config-get vip)
-      nova_url="http://$address:18774/v1.1/\$(tenant_id)s"
-      ec2_url="http://$address:18773/services/Cloud"
-      s3_url="http://$address:13333"
+      local address=$(config-get vip)
+      local region=$(config-get region)
+      local nova_url="$scheme://$address:8774/v1.1/\$(tenant_id)s"
+      local ec2_url="$scheme://$address:8773/services/Cloud"
+      local s3_url="$scheme://$address:3333"
+      local quantum_url="$scheme://$address:9696"
+      local nova_vol_url="$scheme://$address:8776/v1/\$(tenant_id)s"
+
       relation-set -r $r_id \
+        nova_service="nova" \
+        nova_region="$region" \
         nova_public_url="$nova_url" \
         nova_admin_url="$nova_url" \
         nova_internal_url="$nova_url" \
+        ec2_service="ec2" \
+        ec2_region="$region" \
         ec2_public_url="$ec2_url" \
         ec2_admin_url="$ec2_url" \
         ec2_internal_url="$ec2_url" \
+        s3_service="s3" \
+        s3_region="$region" \
         s3_public_url="$s3_url" \
         s3_admin_url="$s3_url" \
         s3_internal_url="$s3_url"
 
       if [ "$(config-get network-manager)" == "Quantum" ]; then
-        quantum_url="http://$address:19696"
         relation-set -r $r_id \
+          quantum_service="quantum" \
+          quantum_region="$region" \
           quantum_public_url="$quantum_url" \
           quantum_admin_url="$quantum_url" \
           quantum_internal_url="$quantum_url"
       fi
 
       if [[ -n "$(relation-ids nova-volume-service)" ]] ; then
-        nova_vol_url="http://$address:18776/v1/\$(tenant_id)s"
         relation-set -r $r_id \
+          nova-volume_service="nova-volume" \
+          nova-volume_region="$region" \
           nova-volume_public_url="$nova_vol_url" \
           nova-volume_admin_url="$nova_vol_url" \
           nova-volume_internal_url="$nova_vol_url"
       fi
     done
     if [ "$(config-get network-manager)" == "Quantum" ]; then
-      # Let gateway nodes use the new HA address for the 
+      # Let gateway nodes use the new HA address for the
       # quantum API server
       for r_id in `relation-ids quantum-network-service`; do
         relation-set -r $r_id \
-          quantum_host=$address
-          quantum_port=19696
+          quantum_url="$quantum_url" region="$region"
       done
     fi
   fi

=== modified file 'revision'
--- revision	2013-03-05 01:41:05 +0000
+++ revision	2013-03-09 04:30:28 +0000
@@ -1,1 +1,1 @@
-214
+236