Merge lp:~egoist-dev/charms/precise/mongodb/replica-set into lp:charms/mongodb

[Bartek Zurawski]

Fixed using mongodb replica-set by this charms. Change way to create replica-set for rs.initiate(configuration) instead of rs.initiate(). Also fix problem with adding and removing members from existed replica set.

[Stuart Bishop (stub)]

Thanks for this work. It looks like the mongo charm certainly needs a polish.

This branch contains conflict markers, so certainly needs repair as the Python will not compile.

Under what circumstances is it sane to change replicaset_server_dbpath or replicaset_server_logpath? If there are no genuine reasons, the charm should be opinionated and the paths hardcoded, rather than having them as configuration items (which really requires tests to confirm things work with non-default paths).

Is there a reason pymongo needs to be installed via pip, or can we use the package? Many installations will not have network connectivity to pypi, so a package is preferred. Otherwise, at a minimum a configurable URL will need to be added to config.yaml, specifying where the tarball can be downloaded and installed from.

There are some mixed tabs and spaces in hooks.py that should be fixed. Quite a lot of the changes appear to be whitespace, replacing spaces with tabs, so I suspect editor settings have caused problems. For example:

- s.shutdown(socket.SHUT_RDWR)
- juju_log("port_check: %s:%s/%s is open" % (host, port, protocol))
- return(True)
+ s.shutdown(socket.SHUT_RDWR)
+ juju_log("port_check: %s:%s/%s is open" % (host, port, protocol))
+ return(True)

At a minimum, it needs to be consistent. I'm seeing all three acceptable forms of indentation being used in the one file, which is a timebomb (4 space only, mixed 4 space + tab, tab only).

A lot of the helpers here could instead be pulled from charm-helpers, in particular relation_get and relation_set. I suspect they would disappear from this review if the whitespace returned to normal.

This code is broken. relation-get will return the same information consistently throughout the hook (and if it doesn't, you have found a Juju bug).

+ while relation_data is None and current_try < max_tries:
+ time.sleep(wait_for)
+ relation_data = json.loads(subprocess.check_output(relation_cmd_line))

I worry about there being race conditions adding and removing the database from the replica set. It seems that the current replica set configuration is being update, and the entire modified configuration resubmitted. If multiple peer relation-changed or relation-departed hooks are running simultaneously, they will race and we will end up with broken replication configuration. I don't think this can be fixed without juju leadership.

review: Needs Fixing

[Charles Butler (lazypower)]

Does not look good to me. Please fix this branch and resubmit.

review: Needs Resubmitting

Unmerged revisions

48. By Bartek Zurawski on 2014-07-28

Fixing replica-set hooks. Using rs.initiate(configuration) instead of rs.initiate() also using rs.conf and rs.reconfig instead of rs.add, and fix problem with adding and removing members from existed replica

=== modified file 'config.yaml'
--- config.yaml	2014-04-11 21:00:49 +0000
+++ config.yaml	2014-07-28 14:22:50 +0000
@@ -108,9 +108,17 @@
     type: string
     description: Size limit for in-memory storage of op ids
   replicaset:
-    default: myset
+    default: "myset"
     type: string
     description: Name of the replica set
+  replicaset_server_dbpath:
+    default: "/mnt/var/lib/database/"
+    type: string
+    description: Path to Database in replica set
+  replicaset_server_logpath:
+    default: "/var/log/mongodb/replica.log"
+    type: string
+    description: Path to logfile in replica set
   web_admin_ui:
     default: True
     type: boolean
@@ -132,7 +140,7 @@
     type: string
     description: The path where the config server data files will be kept.
   config_server_logpath:
-    default: "/mnt/var/log/mongodb/configsvr.log"
+    default: "/var/log/mongodb/configsvr.log"
     type: string
     description: The path where to send config server log data.
   arbiter:
@@ -140,11 +148,11 @@
     type: string
     description: Enable arbiter mode. Possible values are 'disabled' for no arbiter, 'enable' to become an arbiter or 'host:port' to declare another host as an arbiter.  replicaset_master must be set for this option to work.
   mongos_logpath:
-    default: "/mnt/var/log/mongodb/mongos.log"
+    default: "/var/log/mongodb/mongos.log"
     type: string
     description: The path where to send log data from the mongo router.
   mongos_port:
-    default: 27021
+    default: 27017
     type: int
     description: Port number to use for the mongo router
   extra_config_options:

=== modified symlink 'hooks/config-changed'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/database-relation-joined'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2014-07-21 19:41:19 +0000
+++ hooks/hooks.py	2014-07-28 14:22:50 +0000
@@ -16,6 +16,7 @@
 import time
 import yaml
 import argparse
+import pymongo
 
 from os import chmod
 from os import remove
@@ -23,6 +24,8 @@
 from string import Template
 from textwrap import dedent
 from yaml.constructor import ConstructorError
+from random import randint 
+from pymongo import *
 
 from charmhelpers.fetch import (
     add_source,
@@ -40,8 +43,8 @@
 default_mongodb_config = "/etc/mongodb.conf"
 default_mongodb_init_config = "/etc/init/mongodb.conf"
 default_mongos_list = "/etc/mongos.list"
-default_wait_for = 20
-default_max_tries = 20
+default_wait_for = 4
+default_max_tries = 5
 
 ###############################################################################
 # Supporting functions
@@ -129,6 +132,7 @@
 #                              to the specified unit
 #                relation_id:  specify relation id for out of context usage.
 #------------------------------------------------------------------------------
+
 def relation_get(scope=None, unit_name=None, relation_id=None,
         wait_for=default_wait_for, max_tries=default_max_tries):
     juju_log("relation_get: scope: %s, unit_name: %s, relation_id: %s" %
@@ -146,10 +150,9 @@
             relation_cmd_line.append(unit_name)
         relation_data = json.loads(subprocess.check_output(relation_cmd_line))
 
-#        while relation_data is None and current_try < max_tries:
-#            time.sleep(wait_for)
-#            relation_data = json.loads(subprocess.check_output(relation_cmd_line))
-#            current_try += 1
+        while relation_data is None and current_try < max_tries:
+	    time.sleep(wait_for)
+            relation_data = json.loads(subprocess.check_output(relation_cmd_line))
 
     except Exception, e:
         juju_log(str(e))
@@ -158,6 +161,28 @@
         juju_log("relation_get returns: %s" % relation_data)
         return(relation_data)
 
+"""
+def relation_get(attribute=None, unit=None, rid=None):
+	_args = ['relation-get', '--format=json']
+	if rid:
+        	_args.append('-r')
+        	_args.append(rid)
+    	_args.append(attribute or '-')
+	if unit:
+		_args.append(unit)
+	if attribute:
+		_args.append(attribute)
+	try:
+		return json.loads(subprocess.check_output(_args))
+	except ValueError:
+		return None
+	except CalledProcessError, e:
+		if e.returncode == 2:
+			return None
+		raise
+
+"""
+
 
 #------------------------------------------------------------------------------
 # relation_set:  Convenience function wrapping the juju command relation-set
@@ -168,6 +193,7 @@
 #                relation_id:  The relation id to use
 #                Returns: True on success or False on failure
 #------------------------------------------------------------------------------
+
 def relation_set(key_value_pairs=None, relation_id=None):
     juju_log("relation_set: kv: %s, relation_id: %s" %
     (key_value_pairs, relation_id))
@@ -188,7 +214,22 @@
         juju_log("relation_set returns: %s" % retVal)
         return(retVal)
 
-
+"""
+def relation_set(relation_settings={}, relation_id=None):
+	juju_log("Relation Set...")
+	relation_cmd_line = ['relation-set']
+	if relation_id is not None:
+		relation_cmd_line.extend(('-r', relation_id))
+	for (k, v) in relation_settings.items():
+		if v is None:
+			relation_cmd_line.append('{}='.format(k))
+		else:
+			relation_cmd_line.append('{}={}'.format(k, v))
+	try:
+		subprocess.call(relation_cmd_line)
+	except Exception, e:
+		juju_log(str(e))
+"""
 def relation_list(relation_id=None, wait_for=default_wait_for,
         max_tries=default_max_tries):
     juju_log("relation_list: relation_id: %s" % relation_id)
@@ -199,10 +240,11 @@
             relation_cmd_line.append('-r %s' % relation_id)
         relation_data = json.loads(subprocess.check_output(relation_cmd_line))
 
-#        while relation_data is None and current_try < max_tries:
-#            time.sleep(wait_for)
-#            relation_data = json.loads(subprocess.check_output(relation_cmd_line))
-#            current_try += 1
+        while relation_data is None and current_try < max_tries:
+            time.sleep(wait_for)
+	    juju_log("Sleeping...")
+            relation_data = json.loads(subprocess.check_output(relation_cmd_line))
+            current_try += 1
 
     except Exception, e:
         juju_log(str(e))
@@ -255,9 +297,9 @@
         return(False)
     try:
         s.connect((host, int(port)))
-        s.shutdown(socket.SHUT_RDWR)
-        juju_log("port_check: %s:%s/%s is open" % (host, port, protocol))
-        return(True)
+       	s.shutdown(socket.SHUT_RDWR)
+       	juju_log("port_check: %s:%s/%s is open" % (host, port, protocol))
+       	return(True)
     except Exception, e:
         juju_log("port_check: Unable to connect to %s:%s/%s." %
         (host, port, protocol))
@@ -519,13 +561,78 @@
         return(subprocess.call(cmd_line, shell=True) == 0)
 
 
-def init_replset(master_node=None):
+
+def init_replset(master_node=None, hosts=None, replSet=None, wait_for=2, max_tries=default_max_tries):
     if master_node is None:
-        juju_log("init_replset: master_node must be defined.")
-        retVal = False
-    else:
-        retVal = mongo_client(master_node, 'rs.initiate()')
-    juju_log("init_replset returns: %s" % retVal)
+	juju_log("init_replset: master_node is not defined")
+	retVal = False
+    if hosts is None:
+	juju_log("init_replset: hosts list is not defined")
+	retVal = False
+    if replSet is None:
+	juju_log("init_replset: replica set name is not defined")
+	retVal = False
+    command = None
+    current_state = 0 
+    for ip in range(len(hosts)):
+	index = hosts.index(hosts[ip])
+	address = hosts[index]
+	address_split = address.split(':')
+	while (not port_check(address_split[0], address_split[1]) and current_state < max_tries):
+		current_state += 1
+    master_host = master_node.split(':')[0]
+    master_port = master_node.split(':')[1]
+    juju_log("check replica set -> is ok?")
+    
+    if is_replica_set(master_host, master_port):
+	juju_log("replica set -> existed")
+   	c = pymongo.connection.Connection('%s:%s' % (master_host, master_port))
+	mongo = c['admin']
+	local = c['local']['system.replset']
+	cfg = local.find()[0]
+	max_id = 0
+	replica_id = str(cfg['_id'])
+
+	new_cfg = {
+		'_id': replica_id,
+		'members': [],
+	}
+
+	for member in cfg['members']:
+		if max_id < member['_id']:
+			max_id = member['_id']
+
+	for member in cfg['members']:
+		new_cfg['members'].append({'_id': int(member['_id']), 'host': str(member['host'])})
+	for member in hosts:
+		member_host = member.split(':')[0]
+		member_port = member.split(':')[1]
+		if is_replica_set(member_host, member_port) is False:
+			max_id = max_id + 1
+			new_cfg['members'].append({'_id': max_id, 'host': str(member)})
+	new_cfg['version'] = int(cfg['version']) + 1
+	
+	try:
+		juju_log("New replica configuration: %s" % new_cfg)
+		mongo.command({'replSetReconfig': new_cfg})
+	except Exception, e:
+		juju_log(str(e))
+	
+    else:
+    	config = '{"_id":"%s","members":[' % replSet
+    	for ip in range(len(hosts)):
+		index = hosts.index(hosts[ip])
+		config += '{"_id":%s,"host":"%s"}' % (index,hosts[index])
+		if ip+1 != len(hosts):
+			config += ','
+	config += ']}'
+    	command = 'rs.initiate(%s)' % config
+
+    if command:
+    	retVal = mongo_client(master_node, command)
+    else:
+	retVal = True
+    
     return(retVal)
 
 
@@ -548,7 +655,7 @@
         if re.search(' --replSet %s ' % replicaset_name,
                      mongodb_init_config, re.MULTILINE) is None:
             mongodb_init_config = regex_sub([(' -- ',
-                            ' -- --replSet %s ' % replicaset_name)],
+                            ' --replSet %s ' % replicaset_name)],
                             mongodb_init_config)
         retVal = update_file(default_mongodb_init_config, mongodb_init_config)
     except Exception, e:
@@ -557,6 +664,68 @@
     finally:
         return(retVal)
 
+def is_replica_set(host=None, port=None):
+	try:
+		connection = MongoClient('%s' % host, int(port))
+		data = connection.admin.command("replSetGetStatus")
+		if data['ok'] == 1.0:
+			value = True
+		else:
+			value = False
+	except Exception, e:
+		juju_log(str(e))
+		value = False
+	finally:
+		return(value)
+
+def get_replica_set_master(host=None, port=None):
+	replica_master = None
+	try:
+		connection = MongoClient('%s' % host,int(port))
+		data = connection.admin.command("isMaster")
+		if data['ismaster'] is True:
+			replica_master = data['me']
+		elif data['ismaster'] is False and data['setName']:
+			replica_master = data['primary']
+		else:
+			replica_master = None
+	except Exception, e:
+		juju_log(str(e))
+		replica_master = None
+	finally:
+		return(replica_master)
+
+def get_replica_set_name(host=None, port=None):
+	replica_name = None
+	connection = MongoClient('%s' % host, int(port))
+	try:
+		data = connection.admin.command("isMaster")
+		replica_name = data['setName']
+	except Exception, e:
+		juju_log(str(e))
+		replica_name = False
+	finally:
+		return(replica_name)
+
+def is_shard(mongos_host=None, mongos_port=None, client_host=None):
+	connection = MongoClient('%s' % mongos_host,int(mongos_port))
+	shards = connection.admin.command("listshards")['shards']
+	if shards:
+		list_of_shards = shards[0]['host'].split(',')
+		if is_replica_set(client_host,mongos_port) is True:
+			replica_master = get_replica_set_master(client_host, mongos_port)
+			replica_name = get_replica_set_name(client_host, mongos_port)
+			replica_uri = replica_name + '/' + replica_master
+			for member in list_of_shards:
+				if (member == replica_uri or member == replica_master):
+					return True
+		elif is_replica_set(client_host,mongos_port) is False:
+			for member in list_of_shards:
+				client_host2 = client_host + ':' + str(mongos_port)
+				if member == client_host2:
+					return True
+	else:
+		return False
 
 def update_daemon_options(daemon_options=None):
     mongodb_init_config = open(default_mongodb_init_config).read()
@@ -705,9 +874,8 @@
         )
     subprocess.call(
         [
-            'mkdir',
-            '-p',
-            '%s' % os.path.dirname(config_data['config_server_logpath'])
+            'touch',
+            '%s' % config_data['config_server_logpath']
         ]
         )
 
@@ -721,7 +889,8 @@
     cmd_line += " --pidfilepath /var/run/mongodb/configsvr.pid"
     cmd_line += " --fork"
     subprocess.call(cmd_line, shell=True)
-
+    juju_log("Wait for port listening...")
+    time.sleep(wait_for)
     retVal = configsvr_ready(wait_for, max_tries)
     if retVal:
         open_port(config_data['config_server_port'])
@@ -788,26 +957,31 @@
         juju_log("enable_mongos: Not enough config servers yet...")
         return(True)
     disable_mongos()
-    # Make sure logpath exist
-    subprocess.call(
-        [
-            'mkdir',
-            '-p',
-            '%s' % os.path.dirname(config_data['mongos_logpath'])
-        ]
-        )
-    cmd_line = "mongos"
-    cmd_line += " --logpath %s" % config_data['mongos_logpath']
-    cmd_line += " --pidfilepath /var/run/mongodb/mongos.pid"
-    cmd_line += " --port %d" % config_data['mongos_port']
-    cmd_line += " --fork"
-    if len(config_servers) > 0:
-        if len(config_servers) >= 3:
-            cmd_line += ' --configdb %s' % ','.join(config_servers[0:3])
-#        else:
-#            cmd_line += ' --configdb %s' % config_servers[0]
-    juju_log("enable_mongos: cmd_line: %s" % cmd_line)
-    subprocess.call(cmd_line, shell=True)
+    # Stopping mongo to free port 27017
+    if len(config_servers) >=3:
+	service('mongodb', 'stop')	
+	juju_log("Stopping mongo to free port 27017")
+    	# Make sure logpath exist
+    	subprocess.call(
+        	[
+        	    'touch',
+        	    '%s' % config_data['mongos_logpath']
+        	]
+        	)
+    	cmd_line = "mongos"
+    	cmd_line += " --logpath %s" % config_data['mongos_logpath']
+    	cmd_line += " --pidfilepath /var/run/mongodb/mongos.pid"
+    	cmd_line += " --port %d" % config_data['mongos_port']
+    	cmd_line += " --fork"
+    	if len(config_servers) > 0:
+    	    if len(config_servers) >= 3:
+    	        cmd_line += ' --configdb %s' % ','.join(config_servers[0:3])
+#   	     else:
+#       	     cmd_line += ' --configdb %s' % config_servers[0]
+    	juju_log("enable_mongos: cmd_line: %s" % cmd_line)
+	juju_log("Waiting for configsvr")
+	time.sleep(10)    
+	subprocess.call(cmd_line, shell=True)
     retVal = mongos_ready(wait_for, max_tries)
     if retVal:
         open_port(config_data['mongos_port'])
@@ -848,6 +1022,47 @@
         (service('mongodb', 'status') == port_check(my_hostname, my_port))
          is True)
 
+def restart_mongo_replica_config(wait_for=default_wait_for, max_tries=default_max_tries):
+	my_hostname = unit_get('public-address')
+	my_port = config_get('port')
+	current_try = 0
+
+	service('mongodb', 'stop')
+	if os.path.exists('/var/lib/mongodb/mongod.lock'):
+        	os.remove('/var/lib/mongodb/mongod.lock')
+
+	# Make sure dbpath and logpath exist
+	subprocess.call(
+            [
+                'mkdir',
+                '-p',
+                '%s' % config_get('replicaset_server_dbpath')
+            ]
+            )
+        subprocess.call(
+            [
+                'touch',
+                '%s' % config_get('replicaset_server_logpath')
+            ]
+            )
+
+
+	cmd_line = "mongod"
+	cmd_line += " --replSet %s " % config_get('replicaset')
+	cmd_line += " --dbpath %s " % config_get('replicaset_server_dbpath')
+	cmd_line += " --logpath %s " % config_get('replicaset_server_logpath')
+	cmd_line += " --port 27017 "
+	cmd_line += " --fork "
+	cmd_line += " --pidfilepath /var/run/mongodb/replica.pid"
+
+	subprocess.call(cmd_line, shell=True)
+
+	juju_log("Replica waits for config final")
+	while (not port_check(my_hostname, my_port)):
+		time.sleep(wait_for)
+
+	return (port_check(my_hostname, my_port) is True)
+
 
 def backup_cronjob(disable=False):
     """Generate the cronjob to backup with mongodbump."""
@@ -1054,9 +1269,13 @@
 def stop_hook():
     juju_log("stop_hook")
     try:
-        retVal = service('mongodb', 'stop')
-        os.remove('/var/lib/mongodb/mongod.lock')
-        #FIXME Need to check if this is still needed
+	proc = subprocess.Popen(['ps', '-A'], stdout=subprocess.PIPE)
+	out, err = proc.communicate()
+	for line in out.splitlines():
+		if 'mongod' in line:
+			pid = int(line.split(None, 1)[0])
+			os.kill(pid, signal.SIGKILL)
+	retVal = True
     except Exception, e:
         juju_log(str(e))
         retVal = False
@@ -1067,17 +1286,23 @@
 
 def database_relation_joined():
     juju_log("database_relation_joined")
-    my_hostname = unit_get('public-address')
-    my_port = config_get('port')
-    my_replset = config_get('replicaset')
-    juju_log("my_hostname: %s" % my_hostname)
-    juju_log("my_port: %s" % my_port)
-    juju_log("my_replset: %s" % my_replset)
+    hostname = unit_get('public-address')
+    port = config_get('port')
+    replset = config_get('replicaset')
+    install_order = os.environ['JUJU_UNIT_NAME'].split('/')[1]
+    if is_replica_set(hostname, port):
+	replica_master = get_replica_set_master(hostname, port)
+	hostname = replica_master.split(':')[0]
+	port = replica_master.split(':')[1]
+    juju_log("hostname: %s" % hostname)
+    juju_log("port: %s" % port)
+    juju_log("replset: %s" % replset)
     return(relation_set(
         {
-            'hostname': my_hostname,
-            'port': my_port,
-            'replset': my_replset,
+            'hostname': hostname,
+            'port': port,
+            'replset': replset,
+	    'install-order': install_order,
             'type': 'database',
         }))
 
@@ -1093,7 +1318,7 @@
     juju_log("my_replset: %s" % my_replset)
     juju_log("my_install_order: %s" % my_install_order)
     return(enable_replset(my_replset) ==
-    restart_mongod() ==
+    restart_mongo_replica_config() == 
     relation_set(
         {
             'hostname': my_hostname,
@@ -1103,61 +1328,104 @@
             'type': 'replset',
         }))
 
-
 def replica_set_relation_changed():
     juju_log("replica_set_relation_changed")
     my_hostname = unit_get('public-address')
     my_port = config_get('port')
     my_install_order = os.environ['JUJU_UNIT_NAME'].split('/')[1]
+    my_replicaset = config_get('replicaset')
     my_replicaset_master = config_get('replicaset_master')
 
     # If we are joining an existing replicaset cluster, just join and leave.
-    if my_replicaset_master != "auto":
-        return(join_replset(my_replicaset_master, my_hostname))
+#    if my_replicaset_master != "auto":
+#        return(join_replset(my_replicaset_master, my_hostname))
 
     # Default to this node being the master
     master_hostname = my_hostname
     master_port = my_port
     master_install_order = my_install_order
-
+    master_replicaset = my_replicaset
+
+
+    relList = relation_list()
+    while (len(relList) != 1):
     # Check the nodes in the relation to find the master
-    for member in relation_list():
-        juju_log("replica_set_relation_changed: member: %s" % member)
-        hostname = relation_get('hostname', member)
-        port = relation_get('port', member)
-        install_order = relation_get('install-order', member)
-        juju_log("replica_set_relation_changed: install_order: %s" % install_order)
-        if install_order is None:
-            juju_log("replica_set_relation_changed: install_order is None.  relation is not ready")
-            break
-        if int(install_order) < int(master_install_order):
-            master_hostname = hostname
-            master_port = port
-            master_install_order = install_order
+    	for member in relation_list():
+    	    juju_log("replica_set_relation_changed: member: %s" % member)
+       	    hostname = relation_get('hostname', member)
+            port = relation_get('port', member)
+            install_order = relation_get('install-order', member)
+            juju_log("replica_set_relation_changed: install_order: %s" % install_order)
+            if install_order is None:
+                juju_log("replica_set_relation_changed: install_order is None.  relation is not ready")
+                break
+            if int(install_order) < int(master_install_order):
+		juju_log("in IF")
+                master_hostname = hostname
+            	master_port = port
+            	master_install_order = install_order
+
+    	hosts = []
+    	hosts.append("%s:%s" % (my_hostname, my_port))
+
+    	for member in relation_list():
+		juju_log("Getting ip for replica config")
+		hostname = relation_get('hostname', member)
+		port = relation_get('port', member)
+		hosts.append("%s:%s" % (hostname,port)) 
 
     # Initiate the replset
-    init_replset("%s:%s" % (master_hostname, master_port))
+    	if (init_replset("%s:%s" % (master_hostname, master_port), hosts, my_replicaset) is True):
+		break
 
     # Add the rest of the nodes to the replset
-    for member in relation_list():
-        hostname = relation_get('hostname', member)
-        port = relation_get('port', member)
-        if master_hostname != hostname:
-            if hostname == my_hostname:
-                subprocess.call(['mongo',
-                    '--eval',
-                    "rs.add(\"%s\")" % hostname])
-            else:
-                join_replset("%s:%s" % (master_hostname, master_port),
-                    "%s:%s" % (hostname, port))
-
+#    for member in relation_list():
+#        hostname = relation_get('hostname', member)
+#        port = relation_get('port', member)
+#        if master_hostname != hostname:
+#            if hostname == my_hostname:
+#                subprocess.call(['mongo',
+#                    '--eval',
+#                    "rs.add(\"%s\")" % hostname])
+#            else:
+#                join_replset("%s:%s" % (master_hostname, master_port),
+#                    "%s:%s" % (hostname, port))
+#
     # Add this node to the replset ( if needed )
-    if master_hostname != my_hostname:
-        join_replset("%s:%s" % (master_hostname, master_port),
-            "%s:%s" % (my_hostname, my_port))
+#    if master_hostname != my_hostname:
+#        join_replset("%s:%s" % (master_hostname, master_port),
+#            "%s:%s" % (my_hostname, my_port))
 
     return(True)
 
+def replica_set_relation_departed():
+	host = unit_get('public-address')
+	port = config_get('port')
+	replica_member_host = '%s:%s' % (host,port)
+	replica_master = get_replica_set_master(host, port)
+	if replica_master:
+		c = pymongo.connection.Connection('%s' % replica_master)
+		mongo = c['admin']
+		local = c['local']['system.replset']
+		cfg = local.find()[0]
+		for i in range(0, len(cfg['members'])):
+			juju_log("In for statement")
+			if str(cfg['members'][i]['host']) == str(replica_member_host):
+				juju_log('in if statement')
+				del cfg['members'][i]
+		new_cfg = {
+			'_id': str(cfg['_id']),
+			'members': [],
+		}
+		for member in cfg['members']:
+			new_cfg['members'].append({'_id': int(member['_id']), 'host': str(member['host'])})
+		new_cfg['version'] = int(cfg['version']) + 1
+		try:
+			juju_log("New replica config: %s" % new_cfg)
+			mongo.command({'replSetReconfig': new_cfg})
+		except Exception, e:
+			juju_log(str(e))
+	return(True)
 
 def configsvr_relation_joined():
     juju_log("configsvr_relation_joined")
@@ -1169,7 +1437,7 @@
             'hostname': my_hostname,
             'port': my_port,
             'install-order': my_install_order,
-            'type': 'configsvr',
+            'type': 'configsvr'
         }))
 
 
@@ -1201,13 +1469,25 @@
     juju_log("mongos_relation_changed")
     config_data = config_get()
     retVal = False
+    juju_log("Waiting for instance")
+    time.sleep(5)
     for member in relation_list():
         hostname = relation_get('hostname', member)
         port = relation_get('port', member)
         rel_type = relation_get('type', member)
-        if hostname is None or port is None or rel_type is None:
-            juju_log("mongos_relation_changed: relation data not ready.")
-            break
+	install_order = relation_get('install-order')
+#        if hostname is None or port is None or rel_type is None:
+#            juju_log("mongos_relation_changed: relation data not ready.")
+#            break
+	while hostname is None or port is None or rel_type is None:
+		juju_log("Waiting for relation data")
+		time.sleep(5)
+		hostname = relation_get('hostname', member)
+		juju_log("Hostname: %s" % hostname)
+	        port = relation_get('port', member)
+		juju_log("Port: %s" % port)
+	        rel_type = relation_get('type', member)
+		juju_log("Relation type: %s" % rel_type)
         if rel_type == 'configsvr':
             config_servers = load_config_servers(default_mongos_list)
             print "Adding config server: %s:%s" % (hostname, port)
@@ -1227,6 +1507,7 @@
                 mongos_host = "%s:%s" % (
                     unit_get('public-address'),
                     config_get('mongos_port'))
+<<<<<<< TREE
                 shard_command1 = "sh.addShard(\"%s:%s\")" % (hostname, port)
                 retVal1 = mongo_client(mongos_host, shard_command1)
                 replicaset = relation_get('replset', member)
@@ -1237,9 +1518,25 @@
             else:
                 juju_log("Not enough config server for mongos yet.")
                 retVal = True
+=======
+	    if is_shard(unit_get('public-address'),config_get('mongos_port'),hostname) is False:
+		juju_log("Host not in shard...")
+	    	if is_replica_set(hostname, port) is False: 
+			shard_command1 = "sh.addShard(\"%s:%s\")" % (hostname, port)
+        		value = mongo_client(mongos_host, shard_command1)
+        	else:
+			replicaset = get_replica_set_name(hostname, port)
+			replicaset_master = get_replica_set_master(hostname, port)
+	               	shard_command2 = "sh.addShard(\"%s/%s\")" % \
+        	       	(replicaset, replicaset_master)
+                	value = mongo_client(mongos_host, shard_command2)
+                retVal = value is True
+	    else:
+		juju_log("Host in shard...")
+		retVal = True
+>>>>>>> MERGE-SOURCE
         else:
-            juju_log("mongos_relation_change: undefined rel_type: %s" %
-            rel_type)
+            juju_log("mongos_relation_change: undefined rel_type: %s" % rel_type)
             return(False)
     juju_log("mongos_relation_changed returns: %s" % retVal)
     return(retVal)
@@ -1507,6 +1804,8 @@
         retVal = replica_set_relation_joined()
     elif hook_name == "replica-set-relation-changed":
         retVal = replica_set_relation_changed()
+    elif hook_name == "replica-set-relation-departed":
+	retVal = replica_set_relation_departed()
     elif hook_name == "configsvr-relation-joined":
         retVal = configsvr_relation_joined()
     elif hook_name == "configsvr-relation-changed":

=== modified file 'hooks/install'
--- hooks/install	2013-11-25 19:48:00 +0000
+++ hooks/install	2014-07-28 14:22:50 +0000
@@ -1,5 +1,6 @@
 #!/bin/bash
 
-sudo apt-get install "python-yaml"
+sudo apt-get install python-pip -y 
+sudo pip install pymongo
 
 hooks/hooks.py -H install

=== modified symlink 'hooks/mongos-cfg-relation-broken'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/mongos-relation-broken'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/mongos-relation-changed'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/mongos-relation-joined'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/replica-set-relation-changed'
=== target changed u'./hooks.py' => u'hooks.py'
=== added symlink 'hooks/replica-set-relation-departed'
=== target is u'hooks.py'
=== modified symlink 'hooks/replica-set-relation-joined'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/start'
=== target changed u'./hooks.py' => u'hooks.py'
=== modified symlink 'hooks/stop'
=== target changed u'./hooks.py' => u'hooks.py'