Merge lp:~chad.smith/charms/precise/ceph/ceph-with-health into lp:~charmers/charms/precise/ceph/trunk

[Chad Smith]

In the same line as all other openstack services, this branch delivers health_script.d for that will allow landscape-client to validate the local openstack service health during an openstack rolling upgrade.

 Other charms are delivering run-parts health scripts at /var/lib/juju/units/<charm_name>/charm/scripts/health_check.d so landscape client can validate charm service health outside of the juju hooks environment. To support this effort the charm will need to write a /var/lib/juju/units/<charm_name>/charm/scripts/scriptrc file which can establish script environment variables which provide necessary ports, service names or commands for the health_checks.d scripts to run.

ceph service will only be delivering health_checks.d and not add_to_cluster and remove_from_cluster scripts because ceph service is inherently HA and doesn't use corosync services like some of the other charms. If we need to disable a ceph node from within the cluster, or stop rings from rebuilding/rebalancing during a unit reboot, we may have to introduce remove_from_cluster add_to_cluster scripts as entry points for those actions.

[Chad Smith (chad.smith)]

a question I have on ceph is that I see a number of other ports for ceph-mon and ceph-osd on our ceph units, but I don't see where those ports are configured for ceph. I'd like to monitor the other active ceph ports, but not sure if we should hardcode those port numbers.

root@inst-012:~# netstat -lnp | grep ceph
tcp 0 0 0.0.0.0:6802 0.0.0.0:* LISTEN 4708/ceph-osd
tcp 0 0 192.168.64.154:6789 0.0.0.0:* LISTEN 4473/ceph-mon
tcp 0 0 0.0.0.0:6800 0.0.0.0:* LISTEN 4708/ceph-osd
tcp 0 0 0.0.0.0:6801 0.0.0.0:* LISTEN 4708/ceph-osd

[Adam Gandelman (gandelman-a)]

Retargetted and merged against lp:~openstack-charmers/charms/precise/swift-storage/ha-support which contains the most recent common code and will let us track this against the other WIP charms focusing on these features.

review: Approve

[Chad Smith (chad.smith)]

Adam's comment above was accidentally made on the ceph branch but it really applies to the swift-storage-with-health branch... copying comment to lp:~chad.smith/charms/precise/swift-storage/swift-storage-with-health and "Merging" that branch. Reopening this branch as it is not in the ceph upstream code yet.

[James Page (james-page)]

Chad

TBH monitoring OSD health would be better done using ceph itself: ceph osd tree give you a hierarchical view of the ODS healty.

Unmerged revisions

64. By Chad Smith on 2013-03-05

more strict netstat port matching

63. By Chad Smith on 2013-02-28

update revision

62. By Chad Smith on 2013-02-28

missing comma

61. By Chad Smith on 2013-02-28

charm health scripts for landscape-client on ceph

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2013-01-11 09:10:40 +0000
+++ hooks/hooks.py	2013-03-05 22:19:22 +0000
@@ -92,6 +92,10 @@
     if ceph.is_bootstrapped():
         ceph.rescan_osd_devices()
 
+    env_vars = {
+        'OPENSTACK_SCRIPT_CEPH_STAT': '"ceph status | grep -q HEALTH_OK"',
+        'OPENSTACK_PORT_CEPH_MON': 6789}
+    utils.save_script_rc(**env_vars)
     utils.juju_log('INFO', 'End config-changed hook.')
 
 

=== modified file 'hooks/utils.py'
--- hooks/utils.py	2013-02-08 11:09:00 +0000
+++ hooks/utils.py	2013-03-05 22:19:22 +0000
@@ -218,3 +218,19 @@
         answers = dns.resolver.query(hostname, 'A')
         if answers:
             return answers[0].address
+
+def save_script_rc(script_path="scripts/scriptrc", **env_vars):
+    """
+    Write an rc file in the charm-delivered directory containing
+    exported environment variables provided by env_vars. Any charm scripts run
+    outside the juju hook environment can source this scriptrc to obtain
+    updated config information necessary to perform health checks or
+    service changes.
+    """
+    unit_name = os.getenv('JUJU_UNIT_NAME').replace('/', '-')
+    juju_rc_path="/var/lib/juju/units/%s/charm/%s" % (unit_name, script_path)
+    with open(juju_rc_path, 'wb') as rc_script:
+        rc_script.write(
+            "#!/bin/bash\n")
+        [rc_script.write('export %s=%s\n' % (u, p))
+         for u, p in env_vars.iteritems() if u != "script_path"]

=== modified file 'revision'
--- revision	2012-12-17 10:22:51 +0000
+++ revision	2013-03-05 22:19:22 +0000
@@ -1,1 +1,1 @@
-91
+93

=== added directory 'scripts'
=== added directory 'scripts/health_checks.d'
=== added file 'scripts/health_checks.d/script_checks'
--- scripts/health_checks.d/script_checks	1970-01-01 00:00:00 +0000
+++ scripts/health_checks.d/script_checks	2013-03-05 22:19:22 +0000
@@ -0,0 +1,15 @@
+#!/usr/bin/python
+# Run any commands given as an OPENSTACK_SCRIPT environment variable
+import os, sys, subprocess
+
+# Pull any environment variable definitions out of the charm's scriptrc
+scriptrc=  "%s/scriptrc" % os.path.dirname(os.path.dirname(sys.argv[0]))
+
+for line in open(scriptrc):
+  if line.startswith("export OPENSTACK_SCRIPT"):
+       # drop any bookend whitespace and double-quotes
+       command = line.split("=")[1].strip().strip('"')
+       p = subprocess.Popen('%s' % command, shell=True)
+       exitcode =  p.wait()
+       if exitcode != 0:
+           exit(exitcode)

=== added file 'scripts/health_checks.d/service_ports_live'
--- scripts/health_checks.d/service_ports_live	1970-01-01 00:00:00 +0000
+++ scripts/health_checks.d/service_ports_live	2013-03-05 22:19:22 +0000
@@ -0,0 +1,13 @@
+#!/bin/bash
+# Validate that service ports are active
+HEALTH_DIR=`dirname $0`
+SCRIPTS_DIR=`dirname $HEALTH_DIR`
+. $SCRIPTS_DIR/scriptrc
+set -e
+
+# Grab any OPENSTACK_PORT* environment variables
+openstack_ports=`env| awk -F '=' '(/OPENSTACK_PORT/){print $2}'`
+for port in $openstack_ports
+do
+    netstat -ln | grep -q ":$port "
+done