Merge lp:~mariosplivalo/charms/trusty/mongodb/replsets-fix-try into lp:charms/trusty/mongodb

[Mario Splivalo]

This merge fixes lp #1379604. Mongodb charm now provides correct information to Ceilometer charm regarding MongoDB replicaset.

It also makes replicaset deploying more robust, fixing lp #1403698.

Also adds some unittest (for the changed code) and amulet tests for deploying replicaset.

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #300 trusty-mongodb for mariosplivalo mp245059
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /tmp/tmpeQOGBT
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9556195/
Build: http://10.245.162.77:8080/job/charm_unit_test/300/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #272 trusty-mongodb for mariosplivalo mp245059
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9556196/
Build: http://10.245.162.77:8080/job/charm_lint_check/272/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #399 trusty-mongodb for mariosplivalo mp245059
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/399/

[Billy Olsen (billy-olsen)]

Hey Mario - added a few comments of my own. A few things should be improved throughout here.

[Jorge Niedbalski (niedbalski)]

Hello Mario,

Thanks for contributing to this charm. Next are my observations regarding to this first iteration on the review:

1) Lint has failures.

tests/03_deploy_replicaset.py:34:1: E302 expected 2 blank lines, found 1
tests/03_deploy_replicaset.py:45:20: F821 undefined name 'sentry_dict'
tests/03_deploy_replicaset.py:46:18: F821 undefined name 'sentry_dict'
tests/03_deploy_replicaset.py:77:1: E265 block comment should start with '# '
tests/03_deploy_replicaset.py:78:1: E265 block comment should start with '# '
tests/03_deploy_replicaset.py:79:1: E265 block comment should start with '# '
make: *** [lint] Error 1

2) Please change the makefile to just cover the hooks package.

 test: .venv
        @echo Starting unit tests...
- .venv/bin/nosetests -s --nologcapture --with-coverage $(EXTRA) unit_tests/
+ .venv/bin/nosetests -s --nologcapture --with-coverage $(EXTRA) --cover-package hooks unit_tests/

3) Please review the inline comments.

review: Needs Fixing

[Mario Splivalo (mariosplivalo)]

Added my comments inline - stuff I haven't commented make perfect sense to me. Thank you, guys, for the inputs!

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #680 trusty-mongodb for mariosplivalo mp245059
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  tests/03_deploy_replicaset.py:84:1: E265 block comment should start with '# '
  make: *** [lint] Error 1

Full lint test output: http://paste.ubuntu.com/9718870/
Build: http://10.245.162.77:8080/job/charm_lint_check/680/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #709 trusty-mongodb for mariosplivalo mp245059
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  FAILED (errors=2)
  make: *** [test] Error 1

Full unit test output: http://paste.ubuntu.com/9718869/
Build: http://10.245.162.77:8080/job/charm_unit_test/709/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #866 trusty-mongodb for mariosplivalo mp245059
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
  ERROR subprocess encountered error code 1
  make: *** [functional_test] Error 1

Full amulet test output: http://paste.ubuntu.com/9719129/
Build: http://10.245.162.77:8080/job/charm_amulet_test/866/

[Jorge Niedbalski (niedbalski)]

Hello Mario,

Aside from the following lint errors:

tests/03_deploy_replicaset.py:39:1: E302 expected 2 blank lines, found 1
tests/03_deploy_replicaset.py:50:20: F821 undefined name 'sentry_dict'
tests/03_deploy_replicaset.py:51:18: F821 undefined name 'sentry_dict'
tests/03_deploy_replicaset.py:82:1: E265 block comment should start with '# '
tests/03_deploy_replicaset.py:83:1: E265 block comment should start with '# '
tests/03_deploy_replicaset.py:84:1: E265 block comment should start with '# '
hooks/hooks.py:17:1: F401 'operator' imported but unused
hooks/hooks.py:33:1: F401 'fatal' imported but unused
hooks/hooks.py:49:1: F401 'local_unit' imported but unused
hooks/hooks.py:96:33: E113 unexpected indentation
make: *** [lint] Error 1

I made a few observations regarding to the code, I hope you can drive those in order to keep reviewing this proposal.

thanks.

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #701 trusty-mongodb for mariosplivalo mp245059
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  unit_tests/test_hooks.py:269:5: E303 too many blank lines (2)
  make: *** [lint] Error 1

Full lint test output: http://paste.ubuntu.com/9720638/
Build: http://10.245.162.77:8080/job/charm_lint_check/701/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #730 trusty-mongodb for mariosplivalo mp245059
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9720639/
Build: http://10.245.162.77:8080/job/charm_unit_test/730/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #887 trusty-mongodb for mariosplivalo mp245059
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
  ERROR subprocess encountered error code 1
  make: *** [functional_test] Error 1

Full amulet test output: http://paste.ubuntu.com/9720732/
Build: http://10.245.162.77:8080/job/charm_amulet_test/887/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #702 trusty-mongodb for mariosplivalo mp245059
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9720948/
Build: http://10.245.162.77:8080/job/charm_lint_check/702/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #731 trusty-mongodb for mariosplivalo mp245059
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9720949/
Build: http://10.245.162.77:8080/job/charm_unit_test/731/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #888 trusty-mongodb for mariosplivalo mp245059
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
  ERROR subprocess encountered error code 1
  make: *** [functional_test] Error 1

Full amulet test output: http://paste.ubuntu.com/9721024/
Build: http://10.245.162.77:8080/job/charm_amulet_test/888/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #704 trusty-mongodb for mariosplivalo mp245059
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9721529/
Build: http://10.245.162.77:8080/job/charm_lint_check/704/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #733 trusty-mongodb for mariosplivalo mp245059
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9721530/
Build: http://10.245.162.77:8080/job/charm_unit_test/733/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #889 trusty-mongodb for mariosplivalo mp245059
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/889/

[Jorge Niedbalski (niedbalski)]

A few more comments on the code. Please review.

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #813 trusty-mongodb for mariosplivalo mp246519
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9755506/
Build: http://10.245.162.77:8080/job/charm_unit_test/813/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #784 trusty-mongodb for mariosplivalo mp246519
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9755507/
Build: http://10.245.162.77:8080/job/charm_lint_check/784/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #969 trusty-mongodb for mariosplivalo mp246519
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/969/

[Ryan Beisner (1chb1n)]

Precise deploy tests fail with this MP with:

# from deployer output
2015-01-15 12:36:24 [DEBUG] deployer.import: Waiting for units before adding relations
2015-01-15 12:36:24 [ERROR] deployer.env: The following units had errors:
   unit: mongodb/0: machine: 9 agent-state: error details: hook failed: "install"
2015-01-15 12:36:24 [INFO] deployer.cli: Deployment stopped. run time: 431.20

# from mongodb juju unit log
2015-01-15 12:35:41 INFO install Setting up python-pymongo (2.1-1ubuntu0.1) ...
2015-01-15 12:35:41 INFO install Traceback (most recent call last):
2015-01-15 12:35:41 INFO install File "/var/lib/juju/agents/unit-mongodb-0/charm/hooks/install", line 38, in <module>
2015-01-15 12:35:41 INFO install from pymongo import MongoClient
2015-01-15 12:35:41 INFO install ImportError: cannot import name MongoClient
2015-01-15 12:35:41 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1

Trusty, Juno deploy tests are happy.

review: Needs Fixing

[Jorge Niedbalski (niedbalski)]

Please make sure to include 'execd_preinstall' on the TO_PATCH list for /home/niedbalski/src/charms/trusty/mongodb/unit_tests/test_hooks.py

======================================================================
ERROR: test_install_hook (mongodb.unit_tests.test_hooks.MongoHooksTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/niedbalski/src/charms/trusty/mongodb/.venv/local/lib/python2.7/site-packages/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "/home/niedbalski/src/charms/trusty/mongodb/unit_tests/test_hooks.py", line 154, in test_install_hook
    hooks.install_hook()
  File "hooks/hooks.py", line 915, in install_hook
    execd_preinstall()
  File "hooks/charmhelpers/payload/execd.py", line 50, in execd_preinstall
    execd_run('charm-pre-install', execd_dir=execd_dir)
  File "hooks/charmhelpers/payload/execd.py", line 38, in execd_run
    for submodule_path in execd_submodule_paths(command, execd_dir):
  File "hooks/charmhelpers/payload/execd.py", line 30, in execd_submodule_paths
    for module_path in execd_module_paths(execd_dir):
  File "hooks/charmhelpers/payload/execd.py", line 16, in execd_module_paths
    execd_dir = default_execd_dir()
  File "hooks/charmhelpers/payload/execd.py", line 10, in default_execd_dir
    return os.path.join(os.environ['CHARM_DIR'], 'exec.d')
  File "/home/niedbalski/src/charms/trusty/mongodb/.venv/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'CHARM_DIR'

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #887 trusty-mongodb for mariosplivalo mp246519
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  FAILED (errors=2)
  make: *** [test] Error 1

Full unit test output: http://paste.ubuntu.com/9788192/
Build: http://10.245.162.77:8080/job/charm_unit_test/887/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #858 trusty-mongodb for mariosplivalo mp246519
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9788195/
Build: http://10.245.162.77:8080/job/charm_lint_check/858/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1082 trusty-mongodb for mariosplivalo mp246519
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/1082/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #860 trusty-mongodb for mariosplivalo mp246956
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9789503/
Build: http://10.245.162.77:8080/job/charm_lint_check/860/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #889 trusty-mongodb for mariosplivalo mp246956
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9789504/
Build: http://10.245.162.77:8080/job/charm_unit_test/889/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1084 trusty-mongodb for mariosplivalo mp246956
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/1084/

[James Page (james-page)]

Mario

Thanks for the merge proposal; however we do have some complexity in what the precise and trusty versions of this charm need to support.

Specifically, the Ubuntu Server Team provides MongoDB from 14.04 as a backport to 12.04 via the Icehouse cloud-archive; so any change in behaviour really needs to be driven by the version of mongodb installed, rather than the Ubuntu release being used.

I'd prefer we did not start using pip to install things by default; if we can't configure replica-sets correctly with the 12.04 version of mongodb, lets just say we can't do that rather than starting to pull in non-distribution software.

review: Needs Fixing

[Mario Splivalo (mariosplivalo)]

The issue with precise is not mongodb version - it is older and things are different but setting up replicaset is pretty much the same on 2.0 (precise) as it is on 2.4 (trusty).

The issue is that on precise pymongo is very old (it doesn't have MongoClient), also, portalocker is not packaged in precise - hence using pip to install those two dependencies.

[Ryan Beisner (1chb1n)]

FYI: mongodb install hook failed on Precise due to pip install timeout.

[Mario Splivalo (mariosplivalo)]

Yup, becuase in ctsstack/serverstack pip can't reach outside world. Will add pip_proxy in config.yaml so that one can set http proxy specific to deployment environment.

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #966 trusty-mongodb for mariosplivalo mp247136
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9805315/
Build: http://10.245.162.77:8080/job/charm_unit_test/966/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #937 trusty-mongodb for mariosplivalo mp247136
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9805317/
Build: http://10.245.162.77:8080/job/charm_lint_check/937/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1159 trusty-mongodb for mariosplivalo mp247136
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/1159/

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-10968-results

review: Needs Fixing (automated testing)

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #978 trusty-mongodb for mariosplivalo mp247238
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  FAILED (errors=2)
  make: *** [test] Error 1

Full unit test output: http://paste.ubuntu.com/9814549/
Build: http://10.245.162.77:8080/job/charm_unit_test/978/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #949 trusty-mongodb for mariosplivalo mp247238
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  hooks/hooks.py:1245:1: W293 blank line contains whitespace
  make: *** [lint] Error 1

Full lint test output: http://paste.ubuntu.com/9814548/
Build: http://10.245.162.77:8080/job/charm_lint_check/949/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1171 trusty-mongodb for mariosplivalo mp247238
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/1171/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #981 trusty-mongodb for mariosplivalo mp247238
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full unit test output: http://paste.ubuntu.com/9820857/
Build: http://10.245.162.77:8080/job/charm_unit_test/981/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #952 trusty-mongodb for mariosplivalo mp247238
    LINT FAIL: lint-test failed

LINT Results (max last 2 lines):
  Storing debug log for failure in /var/lib/jenkins/.pip/pip.log
  make: *** [.venv] Error 1

Full lint test output: http://paste.ubuntu.com/9820858/
Build: http://10.245.162.77:8080/job/charm_lint_check/952/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #1174 trusty-mongodb for mariosplivalo mp247238
    AMULET OK: passed

Build: http://10.245.162.77:8080/job/charm_amulet_test/1174/

[Jorge Niedbalski (niedbalski)]

Hello,

Thanks Mario for submitting this proposal. I have run several tests in order to test the replicaset consistency.

1) 3, 5 , 7 units with juju deploy.
2) 3 units and then incrementally adding new units until 7.
3) 3 units, add 1, remove primary.
4) 3 units, add 1, remove primary, remove primary
4) 3 units, remove primary

All of the before mentioned cases seems to leave the replica set in consistent state, fixing LP: #1370542

I also validated that unit, lint and amulet tests on both series trusty and precise seems to work as expected.

LGTM

review: Approve

[Ryan Beisner (1chb1n)]

FYI, all deploy tests are happy. [P-I, T-I, T-J, U-J] x [Stable & Next] charms.

[Felipe Reyes (freyes)]

Hi Mario,

Good work, checked that deploys OK with 3 and 5 units, also resizing the cluster, all worked.

Best,

review: Approve

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #982 trusty-mongodb for mariosplivalo mp247238
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
  FAILED (errors=2)
  make: *** [test] Error 1

Full unit test output: http://paste.ubuntu.com/9822375/
Build: http://10.245.162.77:8080/job/charm_unit_test/982/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1032 trusty-mongodb for mariosplivalo mp247238
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1032/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #1025 trusty-mongodb for mariosplivalo mp247136
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/1025/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #1033 trusty-mongodb for mariosplivalo mp247136
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/1033/

=== modified file '.bzrignore'
--- .bzrignore	2014-08-21 14:26:30 +0000
+++ .bzrignore	2015-01-22 15:32:18 +0000
@@ -1,4 +1,9 @@
 .git
+.project
+.pydevproject
+.coverage
+.settings
+.venv/
 bin/*
 scripts/charm-helpers-sync.py
 exec.d/*

=== modified file 'README.md'
--- README.md	2014-11-20 15:42:07 +0000
+++ README.md	2015-01-22 15:32:18 +0000
@@ -12,27 +12,20 @@
 
 ## Review the configurable options
 
-The MongoDB charm allows for certain values to be configurable via a config.yaml file. The options provided are extensive, you should [review the options](https://jujucharms.com/fullscreen/search/precise/mongodb-20/?text=mongodb#bws-configuration). 
+The MongoDB charm allows for certain values to be configurable via a config.yaml file. The options provided are extensive, you should [review the options](https://jujucharms.com/fullscreen/search/precise/mongodb-20/?text=mongodb#bws-configuration).
 
-Specifically the following options are important: 
+Specifically the following options are important:
 
 - replicaset
    - ie: myreplicaset
    - Each replicaset has a unique name to distinguish it’s members from other replicasets available in the network.
-   - The default value of myset should be fine for most single cluster scenarios.
+   - The default value of "myset" should be fine for most single cluster scenarios.
 
 - web_admin_ui
    - MongoDB comes with a basic but very informative web user interface that provides health
      and status information on the database node as well as the cluster.
    - The default value of yes will start the Admin web UI on port 28017.
 
-- replicaset_master
-   - If this node is going to be joining an existing replicaset, you can specify a member of that cluster
-     ( preferably the master node ) so we can join the existing replicaset.
-   - The value should be in the form of host[:port]
-   - ie:  hostname ( will connect to hostname on the default port of 27017 )
-   - ie:  hostname:port  ( will connect to hostname on port number <port> )
-
 Most of the options in config.yaml have been modeled after the default configuration file for mongodb (normally in /etc/mongodb.conf) and should be familiar to most mongodb admins.  Each option in this charm have a brief description of what it does.
 
 # Usage
@@ -46,10 +39,13 @@
 
 ## Replica Sets
 
-Deploy the first MongoDB instance
-
-    juju deploy mongodb
-    juju expose mongodb
+### Deploying
+
+Deploy the first two MongoDB instances that will form replicaset:
+
+    juju deployu mongodb -n 2
+
+Deploying three or more units at start can sometimes lead to unexpected race-conditions so it's best to start with two nodes.
 
 Your deployment should look similar to this ( `juju status` ):
 
@@ -91,18 +87,61 @@
 
     juju set mongodb replicaset=<new_replicaset_name>
 
-### Add one more nodes to your replicaset
+### Add one or more nodes to your replicaset
 
     juju add-unit mongodb
-
-
-### Add multiple nodes to your replicaset
-
-    juju add-unit mongodb -n5
-
+    juju add-unit mongodb -n2
 
 We now have a working MongoDB replica-set.
 
+### Caveats
+
+Keep in mind that you need to have odd number of nodes for a properly formed replicaset.
+
+Replicaset can't function with only one available node - shall this happens the remaining node is switched to 'read-only' until at least one of the broken nodes is restored.
+
+More info can be found in MongoDB documentation at [their website](http://docs.mongodb.org/manual/replication/)
+
+### Removing a failed node
+
+Working units can be removed from replica set using 'juju remove-unit' command. If the removing unit is primary it will automatically be stepped down (so thath re-election of new primary is performend) before being removed.
+However, if a unit fails (freezes, gets destroyed and is unbootable), operator needs to manually remove it. The operator would connect to primary unit, and issue rs.remove() for failed unit. Also, operator needs to issue 'juju remove-unit --force' to remove failed unit from juju.
+
+### Recovering from degraded replicaset
+
+If two members go down replicaset is in read-only state. That is because the
+remaining node is in SECONDARY state (it can't get promoted/voted to PRIMARY
+because there is no majority in replicaset). If failed nodes can't be brought
+back to life we need to manually force remaining node to become a primary. Here
+is how:
+
+ 1. connect to the node that's alive
+ 2. start 'mongo', a cli utility
+ 3. upon connecting you'll see that node is SECONDARY
+ 4. display current configuration with:
+      rs.config()
+      - this will show the alive node as well as the nodes that are unreachabble
+
+ 5. store the configuration into some temporary json document:
+      cfg=rs.config()
+
+ 6. change the cfg document so that it's members array contain only the unit
+    that is alive:
+      cfg.members=[cfg.members[0]]
+
+ 7. force reconfiguration of the replicaset:
+      rs.reconfigure(cfg, {force: true})
+
+ 8. wait a few, and press ENTER. You should see that your node becomes PRIMARY.
+
+After this clean up the unavailable machines from juju:
+       juju remove-machine --force XX    ## XX is the machine number
+
+And add more units to form a proper replicaset. (To avoid race conditions it is
+best to add units one by one).
+
+       juju add-unit mongodb
+
 ## Sharding (Scale Out Usage)
 
 According the the MongoDB documentation found on [their website](http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/), one way of deploying a Shard Cluster is as follows:
@@ -127,7 +166,7 @@
       replicaset: configsvr
 
 We'll save this one as `~/mongodb-shard.yaml`.
-  
+
 ### Bootstrap the environment
     juju bootstrap
 
@@ -210,7 +249,7 @@
 
 ## MongoDB Contact Information
 
-- [MongoDB website](http://mongodb.org) 
+- [MongoDB website](http://mongodb.org)
 - [MongoDB documentation](http://www.mongodb.org/display/DOCS/Home)
 - [MongoDB bug tracker](https://jira.mongodb.org/secure/Dashboard.jspa)
 - [MongoDB user mailing list](https://groups.google.com/forum/#!forum/mongodb-user)

=== modified file 'charm-helpers-sync.yaml'
--- charm-helpers-sync.yaml	2015-01-13 17:51:52 +0000
+++ charm-helpers-sync.yaml	2015-01-22 15:32:18 +0000
@@ -4,4 +4,5 @@
     - core
     - fetch
     - contrib.hahelpers.cluster
+    - contrib.python.packages
     - payload.execd

=== modified file 'config.yaml'
--- config.yaml	2014-12-09 22:54:36 +0000
+++ config.yaml	2015-01-22 15:32:18 +0000
@@ -169,7 +169,7 @@
     description: "Number of backups to keep. Keeps one week's worth by default."
   #------------------------------------------------------------------------
   # Legacy volume management (DEPRECATED)
-  #   volume-map, volume-dev_regexp are only used 
+  #   volume-map, volume-dev_regexp are only used
   #   if volume-ephemeral-storage == False
   #------------------------------------------------------------------------
   volume-ephemeral-storage:

=== modified file 'hooks/charmhelpers/contrib/hahelpers/cluster.py'
--- hooks/charmhelpers/contrib/hahelpers/cluster.py	2014-12-09 23:58:57 +0000
+++ hooks/charmhelpers/contrib/hahelpers/cluster.py	2015-01-22 15:32:18 +0000
@@ -13,6 +13,7 @@
 
 import subprocess
 import os
+
 from socket import gethostname as get_unit_hostname
 
 import six
@@ -28,12 +29,19 @@
     WARNING,
     unit_get,
 )
+from charmhelpers.core.decorators import (
+    retry_on_exception,
+)
 
 
 class HAIncompleteConfig(Exception):
     pass
 
 
+class CRMResourceNotFound(Exception):
+    pass
+
+
 def is_elected_leader(resource):
     """
     Returns True if the charm executing this is the elected cluster leader.
@@ -68,24 +76,30 @@
     return False
 
 
-def is_crm_leader(resource):
+@retry_on_exception(5, base_delay=2, exc_type=CRMResourceNotFound)
+def is_crm_leader(resource, retry=False):
     """
     Returns True if the charm calling this is the elected corosync leader,
     as returned by calling the external "crm" command.
+
+    We allow this operation to be retried to avoid the possibility of getting a
+    false negative. See LP #1396246 for more info.
     """
-    cmd = [
-        "crm", "resource",
-        "show", resource
-    ]
+    cmd = ['crm', 'resource', 'show', resource]
     try:
-        status = subprocess.check_output(cmd).decode('UTF-8')
+        status = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        if not isinstance(status, six.text_type):
+            status = six.text_type(status, "utf-8")
     except subprocess.CalledProcessError:
-        return False
-    else:
-        if get_unit_hostname() in status:
-            return True
-        else:
-            return False
+        status = None
+
+    if status and get_unit_hostname() in status:
+        return True
+
+    if status and "resource %s is NOT running" % (resource) in status:
+        raise CRMResourceNotFound("CRM resource %s not found" % (resource))
+
+    return False
 
 
 def is_leader(resource):

=== added directory 'hooks/charmhelpers/contrib/python'
=== added file 'hooks/charmhelpers/contrib/python/__init__.py'
=== added file 'hooks/charmhelpers/contrib/python/packages.py'
--- hooks/charmhelpers/contrib/python/packages.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/contrib/python/packages.py	2015-01-22 15:32:18 +0000
@@ -0,0 +1,80 @@
+#!/usr/bin/env python
+# coding: utf-8
+
+__author__ = "Jorge Niedbalski <[e-mail]>"
+
+from charmhelpers.fetch import apt_install, apt_update
+from charmhelpers.core.hookenv import log
+
+try:
+    from pip import main as pip_execute
+except ImportError:
+    apt_update()
+    apt_install('python-pip')
+    from pip import main as pip_execute
+
+
+def parse_options(given, available):
+    """Given a set of options, check if available"""
+    for key, value in sorted(given.items()):
+        if key in available:
+            yield "--{0}={1}".format(key, value)
+
+
+def pip_install_requirements(requirements, **options):
+    """Install a requirements file """
+    command = ["install"]
+
+    available_options = ('proxy', 'src', 'log', )
+    for option in parse_options(options, available_options):
+        command.append(option)
+
+    command.append("-r {0}".format(requirements))
+    log("Installing from file: {} with options: {}".format(requirements,
+                                                           command))
+    pip_execute(command)
+
+
+def pip_install(package, fatal=False, upgrade=False, **options):
+    """Install a python package"""
+    command = ["install"]
+
+    available_options = ('proxy', 'src', 'log', "index-url", )
+    for option in parse_options(options, available_options):
+        command.append(option)
+
+    if upgrade:
+        command.append('--upgrade')
+
+    if isinstance(package, list):
+        command.extend(package)
+    else:
+        command.append(package)
+
+    log("Installing {} package with options: {}".format(package,
+                                                        command))
+    pip_execute(command)
+
+
+def pip_uninstall(package, **options):
+    """Uninstall a python package"""
+    command = ["uninstall", "-q", "-y"]
+
+    available_options = ('proxy', 'log', )
+    for option in parse_options(options, available_options):
+        command.append(option)
+
+    if isinstance(package, list):
+        command.extend(package)
+    else:
+        command.append(package)
+
+    log("Uninstalling {} package with options: {}".format(package,
+                                                          command))
+    pip_execute(command)
+
+
+def pip_list():
+    """Returns the list of current python installed packages
+    """
+    return pip_execute(["list"])

=== added file 'hooks/charmhelpers/core/decorators.py'
--- hooks/charmhelpers/core/decorators.py	1970-01-01 00:00:00 +0000
+++ hooks/charmhelpers/core/decorators.py	2015-01-22 15:32:18 +0000
@@ -0,0 +1,41 @@
+#
+# Copyright 2014 Canonical Ltd.
+#
+# Authors:
+#  Edward Hope-Morley <[e-mail]>
+#
+
+import time
+
+from charmhelpers.core.hookenv import (
+    log,
+    INFO,
+)
+
+
+def retry_on_exception(num_retries, base_delay=0, exc_type=Exception):
+    """If the decorated function raises exception exc_type, allow num_retries
+    retry attempts before raise the exception.
+    """
+    def _retry_on_exception_inner_1(f):
+        def _retry_on_exception_inner_2(*args, **kwargs):
+            retries = num_retries
+            multiplier = 1
+            while True:
+                try:
+                    return f(*args, **kwargs)
+                except exc_type:
+                    if not retries:
+                        raise
+
+                delay = base_delay * multiplier
+                multiplier += 1
+                log("Retrying '%s' %d more times (delay=%s)" %
+                    (f.__name__, retries, delay), level=INFO)
+                retries -= 1
+                if delay:
+                    time.sleep(delay)
+
+        return _retry_on_exception_inner_2
+
+    return _retry_on_exception_inner_1

=== modified file 'hooks/charmhelpers/core/host.py'
--- hooks/charmhelpers/core/host.py	2014-12-09 23:58:57 +0000
+++ hooks/charmhelpers/core/host.py	2015-01-22 15:32:18 +0000
@@ -162,13 +162,16 @@
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
     realpath = os.path.abspath(path)
-    if os.path.exists(realpath):
-        if force and not os.path.isdir(realpath):
+    path_exists = os.path.exists(realpath)
+    if path_exists and force:
+        if not os.path.isdir(realpath):
             log("Removing non-directory file {} prior to mkdir()".format(path))
             os.unlink(realpath)
-    else:
+            os.makedirs(realpath, perms)
+    elif not path_exists:
         os.makedirs(realpath, perms)
     os.chown(realpath, uid, gid)
+    os.chmod(realpath, perms)
 
 
 def write_file(path, content, owner='root', group='root', perms=0o444):
@@ -404,13 +407,21 @@
         os.chdir(cur)
 
 
-def chownr(path, owner, group):
+def chownr(path, owner, group, follow_links=True):
     uid = pwd.getpwnam(owner).pw_uid
     gid = grp.getgrnam(group).gr_gid
+    if follow_links:
+        chown = os.chown
+    else:
+        chown = os.lchown
 
     for root, dirs, files in os.walk(path):
         for name in dirs + files:
             full = os.path.join(root, name)
             broken_symlink = os.path.lexists(full) and not os.path.exists(full)
             if not broken_symlink:
-                os.chown(full, uid, gid)
+                chown(full, uid, gid)
+
+
+def lchownr(path, owner, group):
+    chownr(path, owner, group, follow_links=False)

=== modified file 'hooks/charmhelpers/core/templating.py'
--- hooks/charmhelpers/core/templating.py	2014-12-09 23:58:57 +0000
+++ hooks/charmhelpers/core/templating.py	2015-01-22 15:32:18 +0000
@@ -48,5 +48,5 @@
                     level=hookenv.ERROR)
         raise e
     content = template.render(context)
-    host.mkdir(os.path.dirname(target))
+    host.mkdir(os.path.dirname(target), owner, group)
     host.write_file(target, content, owner, group, perms)

=== modified file 'hooks/charmhelpers/fetch/__init__.py'
--- hooks/charmhelpers/fetch/__init__.py	2014-12-09 23:58:57 +0000
+++ hooks/charmhelpers/fetch/__init__.py	2015-01-22 15:32:18 +0000
@@ -64,9 +64,16 @@
     'trusty-juno/updates': 'trusty-updates/juno',
     'trusty-updates/juno': 'trusty-updates/juno',
     'juno/proposed': 'trusty-proposed/juno',
-    'juno/proposed': 'trusty-proposed/juno',
     'trusty-juno/proposed': 'trusty-proposed/juno',
     'trusty-proposed/juno': 'trusty-proposed/juno',
+    # Kilo
+    'kilo': 'trusty-updates/kilo',
+    'trusty-kilo': 'trusty-updates/kilo',
+    'trusty-kilo/updates': 'trusty-updates/kilo',
+    'trusty-updates/kilo': 'trusty-updates/kilo',
+    'kilo/proposed': 'trusty-proposed/kilo',
+    'trusty-kilo/proposed': 'trusty-proposed/kilo',
+    'trusty-proposed/kilo': 'trusty-proposed/kilo',
 }
 
 # The order of this list is very important. Handlers should be listed in from

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2015-01-13 17:51:52 +0000
+++ hooks/hooks.py	2015-01-22 15:32:18 +0000
@@ -28,28 +28,51 @@
     apt_install
 )
 
+import json
+
+from charmhelpers.core.host import (
+    service,
+    lsb_release,
+)
+
 from charmhelpers.core.hookenv import (
+    close_port,
     config,
+    open_port,
     unit_get,
     relation_get,
     relation_set,
     relations_of_type,
     relation_id,
     relation_ids,
-    open_port,
-    close_port,
     Hooks,
+    DEBUG,
+    WARNING,
 )
 
 from charmhelpers.core.hookenv import log as juju_log
 
 from charmhelpers.payload.execd import execd_preinstall
 
-from charmhelpers.core.host import (
-    service,
-    lsb_release,
+from charmhelpers.contrib.hahelpers.cluster import (
+    oldest_peer,
+    peer_units
 )
 
+try:
+    from pymongo import Connection
+    from pymongo.errors import OperationFailure
+except ImportError:
+    apt_install("python-pymongo", fatal=True)
+    from pymongo import Connection
+    from pymongo.errors import OperationFailure
+
+try:
+    from lockfile import FileLock
+except ImportError:
+    apt_install("python-lockfile", fatal=True)
+    from lockfile import FileLock
+
 hooks = Hooks()
 
 ###############################################################################
@@ -61,6 +84,33 @@
 default_wait_for = 10
 default_max_tries = 5
 
+INSTALL_PACKAGES = ['mongodb-server', 'python-yaml']
+
+INIT_LOCKFILE = '/tmp/mongodb-charm.lock'
+
+# number of seconds init_replset will pause while looping to check if
+# replicaset is initialized
+INIT_CHECK_DELAY = 1.5
+
+# number of times mongo_client_smart will try to execute given statement
+MONGO_CLIENT_RETRIES = 10
+
+# These are MongoDB ReplicaSet states, for convenience:
+MONGO_STARTUP = 0
+MONGO_PRIMARY = 1
+MONGO_SECONDARY = 2
+MONGO_RECOVERING = 3
+MONGO_FATAL = 4
+MONGO_STARTUP2 = 5
+MONGO_UNKNOWN = 6
+MONGO_ARBITER = 7
+MONGO_DOWN = 8
+MONGO_ROLLBACK = 9
+MONGO_REMOVED = 10
+
+# Ill and quick way to make relatinon-departed and relation-broken communicate
+was_i_primary = False
+
 ###############################################################################
 # Supporting functions
 ###############################################################################
@@ -119,7 +169,7 @@
 
 
 def update_file(filename=None, new_data=None, old_data=None):
-    juju_log("update_file: %s" % filename)
+    juju_log("update_file: %s" % filename, level=DEBUG)
     if filename is None or new_data is None:
         retVal = False
     try:
@@ -131,7 +181,6 @@
         juju_log(str(e))
         retVal = False
     finally:
-        juju_log("update_file %s returns: %s" % (filename, retVal))
         return(retVal)
 
 
@@ -161,6 +210,14 @@
         return((None, None))
 
 
+class MasterNotFoundException(Exception):
+    pass
+
+
+class TimeoutException(Exception):
+    pass
+
+
 ###############################################################################
 # Charm support functions
 ###############################################################################
@@ -336,49 +393,160 @@
     else:
         cmd_line = 'mongo'
         cmd_line += ' --host %s' % host
-        cmd_line += ' --eval \'%s\'' % command
-        juju_log("Executing: %s" % cmd_line)
+        cmd_line += ' --eval \'printjson(%s)\'' % command
+        juju_log("Executing: %s" % cmd_line, level=DEBUG)
         return(subprocess.call(cmd_line, shell=True) == 0)
 
 
-def init_replset(master_node=None):
-    if master_node is None:
-        juju_log("init_replset: master_node must be defined.")
-        retVal = False
-    else:
-        retVal = mongo_client(master_node, 'rs.initiate()')
-    juju_log("init_replset returns: %s" % retVal)
+def mongo_client_smart(host='localhost', command=None):
+    '''
+    Rework of mongo_client function, it retries the command
+    MONGO_CLIENT_RETRIES times
+
+    :param host: The host to connect to. Defaults to localhost
+    :param command: The command to be executed. Can't be None
+    :returns True if command succeeded, False if it failed
+    '''
+
+    if command is None:
+        return False
+
+    cmd_line = ['mongo', '--quiet', '--host', host,
+                '--eval', 'printjson(%s)' % command]
+    juju_log("mongo_client_smart executing: %s" % str(cmd_line), level=DEBUG)
+
+    for i in xrange(MONGO_CLIENT_RETRIES):
+        try:
+            cmd_output = subprocess.check_output(cmd_line)
+            juju_log('mongo_client_smart executed, output: %s' %
+                     cmd_output)
+            if json.loads(cmd_output)['ok'] == 1:
+                return True
+        except subprocess.CalledProcessError as err:
+            juju_log('mongo_client_smart failed: %s' %
+                     err.output,
+                     level=DEBUG)
+            pass
+        finally:
+            time.sleep(1.5)
+
+    # At this point, the command failed
+    juju_log('mongo_client_smart failed executing: %s', level=WARNING)
+    return False
+
+
+def init_replset():
+    config_data = config()
+    # Use my IP at rs.initiate(), voids issues with invalid (and/or
+    # not resolvable by peers) hostnames
+
+    rset = config_data['replicaset']
+    addr = unit_get('private-address')
+    port = config_data['port']
+
+    init = '{_id: "%s", members: [{_id: 0, host: "%s:%s"}]} ' % (rset,
+                                                                 addr,
+                                                                 port)
+
+    retVal = mongo_client('localhost', 'rs.initiate(%s)' % init)
+    time.sleep(1)  # give mongod some time to become primary
+    c = Connection('localhost')
+    while True:
+        try:
+            r = run_admin_command(c, 'replSetGetStatus')
+            mongo_state = r['myState']
+            juju_log('init_replset: myState: %s' % mongo_state)
+            if mongo_state == MONGO_PRIMARY:  # we're primary!
+                break
+            elif mongo_state in (MONGO_STARTUP,
+                                 MONGO_STARTUP2,
+                                 MONGO_SECONDARY
+                                 ):  # we are still initializing
+                continue
+            else:
+                juju_log('init_replset: Unexpected replicaSet state: %s' %
+                         mongo_state)
+                retVal = False
+                break
+        except OperationFailure as e:
+            juju_log('init_replset: OperationFailure: %s' % e)
+            if 'Received replSetInitiate' in str(e):
+                continue
+            else:
+                raise
+        finally:
+            time.sleep(INIT_CHECK_DELAY)
+
+    juju_log("init_replset returns: %s" % retVal, level=DEBUG)
     return(retVal)
 
 
+def run_admin_command(client, cmdstr):
+    """Runs an admin command against the client. Primary purpose is to
+    simplify the unit testing.
+    """
+    return client.admin.command(cmdstr)
+
+
 def join_replset(master_node=None, host=None):
+    # TODO: This methoud shouldn't accept any arguments becase we don't care
+    #       about master_node - we always run replset admin commands on
+    #       localhost.
+    #       However, that might break other code calling this method.
+    #       This will wait charm rewrite.
     juju_log("join_replset: master_node: %s, host: %s" %
              (master_node, host))
     if master_node is None or host is None:
         retVal = False
     else:
-        retVal = mongo_client(master_node, "rs.add(\"%s\")" % host)
-    juju_log("join_replset returns: %s" % retVal)
+        retVal = mongo_client_smart('localhost', 'rs.add("%s")' % host)
+    juju_log("join_replset returns: %s" % retVal, level=DEBUG)
+    return(retVal)
+
+
+def leave_replset(master_node=None, host=None):
+    juju_log("leave_replset: master_node: %s, host: %s" %
+             (master_node, host))
+    if master_node is None or host is None:
+        retVal = False
+    else:
+        retVal = mongo_client(master_node, 'rs.remove("%s")' % host)
+    juju_log("leave_replset returns: %s" % retVal, level=DEBUG)
     return(retVal)
 
 
 def enable_replset(replicaset_name=None):
+    retVal = False
     if replicaset_name is None:
-        retVal = False
+        juju_log('enable_replset: replicaset_name is None, exiting',
+                 level=DEBUG)
     try:
-        mongodb_init_config = open(default_mongodb_init_config).read()
-        if re.search(' --replSet %s ' % replicaset_name,
-                     mongodb_init_config, re.MULTILINE) is None:
-            mongodb_init_config = regex_sub([(' -- ',
-                                              ' -- --replSet %s ' %
-                                              replicaset_name)],
-                                            mongodb_init_config)
-        retVal = update_file(default_mongodb_init_config, mongodb_init_config)
+        juju_log('enable_replset: trying to get lock on: %s' %
+                 default_mongodb_init_config)
+        with FileLock(INIT_LOCKFILE):
+            juju_log('enable_replset: lock acquired', level=DEBUG)
+            with open(default_mongodb_init_config) as mongo_init_file:
+                mongodb_init_config = mongo_init_file.read()
+            if re.search(' --replSet %s ' % replicaset_name,
+                         mongodb_init_config, re.MULTILINE) is None:
+                juju_log('enable_replset: --replset not preset,'
+                         ' enabling',
+                         level=DEBUG)
+                mongodb_init_config = regex_sub([(' -- ',
+                                                  ' -- --replSet %s ' %
+                                                  replicaset_name)],
+                                                mongodb_init_config)
+                update_file(default_mongodb_init_config,
+                            mongodb_init_config)
+                retVal = True
+
+        juju_log('enable_replset will return: %s' % str(retVal), level=DEBUG)
+
     except Exception, e:
-        juju_log(str(e))
+        juju_log(str(e), level=WARNING)
         retVal = False
     finally:
-        return(retVal)
+        return retVal
 
 
 def update_daemon_options(daemon_options=None):
@@ -752,8 +920,7 @@
         arm64_trusty_quirk()
 
     apt_update(fatal=True)
-    apt_install(packages=['mongodb-server', 'mongodb-clients', 'python-yaml'],
-                fatal=True)
+    apt_install(packages=INSTALL_PACKAGES, fatal=True)
 
 
 @hooks.hook('config-changed')
@@ -928,28 +1095,34 @@
     juju_log("my_hostname: %s" % my_hostname)
     juju_log("my_port: %s" % my_port)
     juju_log("my_replset: %s" % my_replset)
-
-    return(relation_set(relation_id(), {
-        'hostname': my_hostname,
-        'port': my_port,
-        'replset': my_replset,
-        'type': 'database',
-    }))
+    relation_data = {'hostname': my_hostname,
+                     'port': my_port,
+                     'type': 'database',
+                     }
+
+    if len(peer_units('replica-set')) > 1:
+        relation_data['replset'] = my_replset
+
+    relation_set(relation_id(), relation_data)
 
 
 @hooks.hook('replicaset-relation-joined')
 def replica_set_relation_joined():
-    juju_log("replica_set_relation_joined")
+    juju_log("replica_set_relation_joined-start")
     my_hostname = unit_get('private-address')
     my_port = config('port')
     my_replset = config('replicaset')
+
     my_install_order = os.environ['JUJU_UNIT_NAME'].split('/')[1]
     juju_log("my_hostname: %s" % my_hostname)
     juju_log("my_port: %s" % my_port)
     juju_log("my_replset: %s" % my_replset)
     juju_log("my_install_order: %s" % my_install_order)
-    enable_replset(my_replset)
-    restart_mongod()
+    # do not restart mongodb if config has not changed
+    if enable_replset(my_replset):
+        juju_log('Restarting mongodb after config change (enable replset)',
+                 level=DEBUG)
+        restart_mongod()
 
     relation_set(relation_id(), {
         'hostname': my_hostname,
@@ -958,61 +1131,118 @@
         'install-order': my_install_order,
         'type': 'replset',
         })
+    juju_log("replica_set_relation_joined-finish")
+
+
+def am_i_primary():
+    c = Connection('localhost')
+    for i in xrange(10):
+        try:
+            r = run_admin_command(c, 'replSetGetStatus')
+            juju_log('am_i_primary: replSetGetStatus returned: %s' % str(r),
+                     level=DEBUG)
+            return r['myState'] == MONGO_PRIMARY
+        except OperationFailure as e:
+            juju_log('am_i_primary: OperationError: %s' % str(e), level=DEBUG)
+            if 'replSetInitiate - should come online shortly' in str(e):
+                # replSet initialization in progress
+                continue
+            elif 'EMPTYCONFIG' in str(e):
+                # replication not
+                return False
+            else:
+                raise
+        finally:
+            time.sleep(1.5)
+
+    # Raise an error if we exhausted the maximum amount of trials
+    raise TimeoutException('Unable to determine if local unit is primary')
 
 
 @hooks.hook('replicaset-relation-changed')
 def replica_set_relation_changed():
-    juju_log("replica_set_relation_changed")
-    my_hostname = unit_get('private-address')
-    my_port = config('port')
-    my_install_order = os.environ['JUJU_UNIT_NAME'].split('/')[1]
-    my_replicaset_master = config('replicaset_master')
-
-    # If we are joining an existing replicaset cluster, just join and leave.
-    if my_replicaset_master != "auto":
-        return(join_replset(my_replicaset_master, my_hostname))
-
-    # Default to this node being the master
-    master_hostname = my_hostname
-    master_port = my_port
-    master_install_order = my_install_order
-
-    # Check the nodes in the relation to find the master
-    for member in relations_of_type('replica-set'):
-        member = member['__unit__']
-        juju_log("replica_set_relation_changed: member: %s" % member)
-        hostname = relation_get('hostname', member)
-        port = relation_get('port', member)
-        inst_ordr = relation_get('install-order', member)
-        juju_log("replica_set_relation_changed: install_order: %s" % inst_ordr)
-        if inst_ordr is None:
-            juju_log("replica_set_relation_changed: install_order is None."
-                     "  relation is not ready")
-            break
-        if int(inst_ordr) < int(master_install_order):
-            master_hostname = hostname
-            master_port = port
-            master_install_order = inst_ordr
-
-    # Initiate the replset
-    init_replset("%s:%s" % (master_hostname, master_port))
-
-    # Add the rest of the nodes to the replset
-    for member in relations_of_type('replica-set'):
-        hostname = relation_get('hostname', member['__unit__'])
-        port = relation_get('port', member['__unit__'])
-        if master_hostname != hostname:
-            if hostname == my_hostname:
-                subprocess.call(['mongo', '--eval',
-                                 "rs.add(\"%s\")" % hostname])
-            else:
-                join_replset("%s:%s" % (master_hostname, master_port),
-                             "%s:%s" % (hostname, port))
-
-    # Add this node to the replset ( if needed )
-    if master_hostname != my_hostname:
-        join_replset("%s:%s" % (master_hostname, master_port),
-                     "%s:%s" % (my_hostname, my_port))
+    private_address = unit_get('private-address')
+    remote_hostname = relation_get('hostname')
+
+    juju_log('replica_set_relation_changed-start')
+    juju_log('local unit: %s, joining_unit: %s' % (private_address,
+                                                   remote_hostname),
+             level=DEBUG)
+
+    if remote_hostname is None:
+        juju_log('Joiner not ready yet... bailing out')
+        return
+
+    # Initialize the replicaset - we do this only on the oldest unit in replset
+    # TODO: figure a way how to avoid race conditions - when unit/1 actually
+    #      comes up before unit/0 does - happens rarely, but can happen
+    # quickfix - deploy replset with only two units, use 'add-unit' to
+    #            add the rest
+    if oldest_peer(peer_units('replica-set')):
+        juju_log('Initializing replicaset')
+        init_replset()
+
+    unit = "%s:%s" % (private_address, config('port'))
+    unit_remote = "%s:%s" % (remote_hostname, relation_get('port'))
+
+    # If this is primary, add joined unit to replicaset
+    if am_i_primary():
+        juju_log('Adding new secondary... %s' % unit_remote, level=DEBUG)
+        join_replset(unit, unit_remote)
+
+    juju_log('replica_set_relation_changed-finish')
+
+
[e-mail]('replicaset-relation-departed')
+def replica_set_relation_departed():
+    juju_log('replica_set_relation_departed-start')
+
+    if not am_i_primary():
+        juju_log('replica_set_relation_departed-finish')
+        return
+
+    unit_address, unit_port = unit_get('private-address'), config('port')
+    remote_address = relation_get('private-address')
+    remote_port = relation_get('port')
+
+    # If I am the unit being removed, step me down from being primary
+    if (unit_address, unit_port) == (remote_address, remote_port):
+        juju_log('Stepping down from being primary...')
+        global was_i_primary
+        was_i_primary = True
+        mongo_client('localhost', 'rs.stepDown()')
+        juju_log('replica_set_relation_departed-finish')
+        return
+
+    unit = "%s:%s" % (unit_get('private-address'),
+                      config('port'))
+    unit_remote = "%s:%s" % (relation_get('hostname'),
+                             relation_get('port'))
+
+    leave_replset(unit, unit_remote)
+    juju_log('Removed %s from replicaset' % unit_remote)
+    juju_log('replica_set_relation_departed-finish')
+
+
[e-mail]('replicaset-relation-broken')
+def replica_set_relation_broken():
+    juju_log('replica_set_relation_broken-start')
+
+    if am_i_primary():
+        juju_log('I was primary - removing myself via new primary.', 'DEBUG')
+        mongo_client('localhost', 'rs.stepDown()')
+        time.sleep(15)   # give some time to for re-election to happen
+
+    c = Connection('localhost')
+    r = c.admin.command('isMaster')
+    master_node = r['primary']
+    unit = "%s:%s" % (unit_get('private-address'),
+                      config('port'))
+
+    juju_log('Removing myself via %s' % (master_node), 'DEBUG')
+    leave_replset(master_node, unit)
+
+    juju_log('replica_set_relation_broken-finish')
 
 
 @hooks.hook('data-relation-joined')

=== added symlink 'hooks/replica-set-relation-broken'
=== target is u'hooks.py'
=== added file 'setup.cfg'
--- setup.cfg	1970-01-01 00:00:00 +0000
+++ setup.cfg	2015-01-22 15:32:18 +0000
@@ -0,0 +1,6 @@
+[nosetests]
+verbosity=2
+with-coverage=1
+cover-erase=1
+cover-package=hooks
+

=== modified file 'test_requirements.txt'
--- test_requirements.txt	2014-12-09 16:08:35 +0000
+++ test_requirements.txt	2015-01-22 15:32:18 +0000
@@ -2,4 +2,4 @@
 mock>=1.0.1
 nose>=1.3.1
 flake8
-
+filelock

=== added file 'tests/00_setup.sh'
--- tests/00_setup.sh	1970-01-01 00:00:00 +0000
+++ tests/00_setup.sh	2015-01-22 15:32:18 +0000
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+set -e
+
+sudo apt-get install python-setuptools -y
+sudo add-apt-repository ppa:juju/stable -y
+
+sudo apt-get update
+sudo apt-get install amulet python3 python3-requests python3-pymongo juju-core charm-tools python-mock python-pymongo -y

=== removed file 'tests/00_setup.sh'
--- tests/00_setup.sh	2014-12-09 15:37:16 +0000
+++ tests/00_setup.sh	1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-#!/bin/bash
-
-set -e
-
-sudo apt-get install python-setuptools -y
-sudo add-apt-repository ppa:juju/stable -y
-
-sudo apt-get update
-sudo apt-get install amulet python3 python3-requests python3-pymongo juju-core charm-tools python-mock python-pymongo -y

=== added file 'tests/03_deploy_replicaset.py'
--- tests/03_deploy_replicaset.py	1970-01-01 00:00:00 +0000
+++ tests/03_deploy_replicaset.py	2015-01-22 15:32:18 +0000
@@ -0,0 +1,114 @@
+#!/usr/bin/env python3
+
+import amulet
+import requests
+import time
+from pymongo import MongoClient
+from collections import Counter
+
+
+#########################################################
+# Test Quick Config
+#########################################################
+scale = 3
+seconds = 900
+
+# amount of time to wait before testing for replicaset
+# status
+wait_for_replicaset = 15
+
+#########################################################
+# 3shard cluster configuration
+#########################################################
+d = amulet.Deployment(series='trusty')
+
+d.add('mongodb', charm='mongodb', units=scale)
+d.expose('mongodb')
+
+# Perform the setup for the deployment.
+try:
+    d.setup(seconds)
+    d.sentry.wait(seconds)
+except amulet.helpers.TimeoutError:
+    message = 'The environment did not setup in %d seconds.', seconds
+    amulet.raise_status(amulet.SKIP, msg=message)
+except:
+    raise
+
+sentry_dict = {
+    'mongodb0-sentry': d.sentry.unit['mongodb/0'],
+    'mongodb1-sentry': d.sentry.unit['mongodb/1'],
+    'mongodb2-sentry': d.sentry.unit['mongodb/2'],
+}
+
+
+#############################################################
+# Check presence of MongoDB GUI HEALTH Status
+#############################################################
+def validate_status_interface():
+    r = requests.get("http://{}:28017".format(
+        d.sentry.unit['mongodb/0'].info['public-address']),
+        verify=False)
+    r.raise_for_status
+
+
+#############################################################
+# Validate that each unit has an active mongo service
+#############################################################
+def validate_running_services():
+    for service in sentry_dict:
+        output = sentry_dict[service].run('service mongodb status')
+        service_active = str(output).find('mongodb start/running')
+        if service_active == -1:
+            message = "Failed to find running MongoDB on host {}".format(
+                service)
+            amulet.raise_status(amulet.SKIP, msg=message)
+
+
+#############################################################
+# Validate proper replicaset setup
+#############################################################
+def validate_replicaset_setup():
+
+    time.sleep(wait_for_replicaset)
+
+    unit_status = []
+
+    for service in sentry_dict:
+        client = MongoClient(sentry_dict[service].info['public-address'])
+        r = client.admin.command('replSetGetStatus')
+        unit_status.append(r['myState'])
+        client.close()
+
+    primaries = Counter(unit_status)[1]
+    if primaries != 1:
+        message = "Only one PRIMARY unit allowed! Found: %s" % (primaries)
+        amulet.raise_status(amulet.FAIL, message)
+
+    secondrs = Counter(unit_status)[2]
+    if secondrs != 2:
+        message = "Only two SECONDARY units allowed! (Found %s)" % (secondrs)
+        amulet.raise_status(amulet.FAIL, message)
+
+
+#############################################################
+# Validate connectivity from $WORLD
+#############################################################
+def validate_world_connectivity():
+    client = MongoClient(d.sentry.unit['mongodb/0'].info['public-address'])
+
+    db = client['test']
+    # Can we successfully insert?
+    insert_id = db.amulet.insert({'assert': True})
+    if insert_id is None:
+        amulet.raise_status(amulet.FAIL, msg="Failed to insert test data")
+    # Can we delete from a shard using the Mongos hub?
+    result = db.amulet.remove(insert_id)
+    if result['err'] is not None:
+        amulet.raise_status(amulet.FAIL, msg="Failed to remove test data")
+
+
+validate_status_interface()
+validate_running_services()
+validate_replicaset_setup()
+validate_world_connectivity()

=== added file 'unit_tests/__init__.py'
--- unit_tests/__init__.py	1970-01-01 00:00:00 +0000
+++ unit_tests/__init__.py	2015-01-22 15:32:18 +0000
@@ -0,0 +1,2 @@
+import sys
+sys.path.append('hooks')

=== added file 'unit_tests/test_hooks.py'
--- unit_tests/test_hooks.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_hooks.py	2015-01-22 15:32:18 +0000
@@ -0,0 +1,341 @@
+from mock import patch, call
+
+import hooks
+
+from test_utils import CharmTestCase
+from pymongo.errors import OperationFailure
+from subprocess import CalledProcessError
+
+# Defines a set of functions to patch on the hooks object. Any of these
+# methods will be patched by default on the default invocations of the
+# hooks.some_func(). Invoking the the interface change relations will cause
+# the hooks context to be created outside of the normal mockery.
+TO_PATCH = [
+    'relation_id',
+    'relation_get',
+    'relation_set',
+    'unit_get',
+    'juju_log',
+    'config',
+]
+
+
+class MongoHooksTest(CharmTestCase):
+
+    def setUp(self):
+        super(MongoHooksTest, self).setUp(hooks, TO_PATCH)
+
+        # The self.config object can be used for direct invocations of the
+        # hooks methods. The side_effect of invoking the config object within
+        # the hooks object will return the value that is set in the test case's
+        # test_config dictionary
+        self.config.side_effect = self.test_config.get
+        self.relation_get.side_effect = self.test_relation.get
+
+    @patch.object(hooks, 'restart_mongod')
+    @patch.object(hooks, 'enable_replset')
+    # Note: patching the os.environ dictionary in-line here so there's no
+    # additional parameter sent into the function
+    @patch.dict('os.environ', JUJU_UNIT_NAME='fake-unit/0')
+    def test_replica_set_relation_joined(self, mock_enable_replset,
+                                         mock_restart):
+        self.unit_get.return_value = 'private.address'
+        self.test_config.set('port', '1234')
+        self.test_config.set('replicaset', 'fake-replicaset')
+        self.relation_id.return_value = 'fake-relation-id'
+
+        mock_enable_replset.return_value = False
+
+        hooks.replica_set_relation_joined()
+
+        # Verify that mongodb was NOT restarted since the replicaset we claimed
+        # was not enabled.
+        self.assertFalse(mock_restart.called)
+
+        exp_rel_vals = {'hostname': 'private.address',
+                        'port': '1234',
+                        'replset': 'fake-replicaset',
+                        'install-order': '0',
+                        'type': 'replset'}
+        # Check that the relation data was set as we expect it to be set.
+        self.relation_set.assert_called_with('fake-relation-id', exp_rel_vals)
+
+        mock_enable_replset.reset_mock()
+        self.relation_set.reset_mock()
+        mock_enable_replset.return_value = True
+
+        hooks.replica_set_relation_joined()
+
+        self.assertTrue(mock_restart.called)
+        self.relation_set.assert_called_with('fake-relation-id', exp_rel_vals)
+
+    @patch.object(hooks, 'run_admin_command')
+    @patch.object(hooks, 'Connection')
+    @patch.object(hooks, 'config')
+    @patch.object(hooks, 'mongo_client')
+    @patch('time.sleep')
+    def test_init_repl_set(self, mock_sleep, mock_mongo_client_fn,
+                           mock_config, mock_mongo_client,
+                           mock_run_admin_command):
+        mock_mongo_client_fn.return_value = False
+
+        mock_config.return_value = {'replicaset': 'foo',
+                                    'private-address': 'mongo.local',
+                                    'port': '12345'}
+
+        # Put the OK state (1) at the end and check the loop.
+        ret_values = [{'myState': x} for x in [0, 2, 5, 1]]
+        mock_run_admin_command.side_effect = ret_values
+
+        hooks.init_replset()
+
+        mock_run_admin_command.assert_called()
+        self.assertEqual(len(ret_values), mock_run_admin_command.call_count)
+        self.assertEqual(len(ret_values) + 1, mock_sleep.call_count)
+
+        mock_run_admin_command.reset_mock()
+        exc = [OperationFailure('Received replSetInitiate'),
+               OperationFailure('unhandled')]
+        mock_run_admin_command.side_effect = exc
+
+        try:
+            hooks.init_replset()
+            self.assertTrue(False, msg="Expected error")
+        except OperationFailure:
+            pass
+
+        mock_run_admin_command.assert_called()
+        self.assertEqual(2, mock_run_admin_command.call_count)
+
+    @patch.object(hooks, 'mongo_client_smart')
+    def test_join_replset(self, mock_mongo_client):
+        hooks.join_replset()
+        self.assertFalse(mock_mongo_client.called)
+
+        mock_mongo_client.reset_mock()
+        hooks.join_replset(master_node='mongo.local')
+        self.assertFalse(mock_mongo_client.called)
+
+        mock_mongo_client.reset_mock()
+        hooks.join_replset(host='fake-host')
+        self.assertFalse(mock_mongo_client.called)
+
+        mock_mongo_client.reset_mock()
+        hooks.join_replset(master_node='mongo.local', host='fake-host')
+        mock_mongo_client.assert_called_with('localhost',
+                                             'rs.add("fake-host")')
+
+    @patch.object(hooks, 'mongo_client')
+    def test_leave_replset(self, mock_mongo_client):
+        hooks.leave_replset()
+        self.assertFalse(mock_mongo_client.called)
+
+        mock_mongo_client.reset_mock()
+        hooks.leave_replset(master_node='mongo.local')
+        self.assertFalse(mock_mongo_client.called)
+
+        mock_mongo_client.reset_mock()
+        hooks.leave_replset(host='fake-host')
+        self.assertFalse(mock_mongo_client.called)
+
+        mock_mongo_client.reset_mock()
+        hooks.leave_replset('mongo.local', 'fake-host')
+        mock_mongo_client.assert_called_with('mongo.local',
+                                             'rs.remove("fake-host")')
+
+    @patch.object(hooks, 'apt_install')
+    @patch.object(hooks, 'apt_update')
+    @patch.object(hooks, 'add_source')
+    @patch.dict('os.environ', CHARM_DIR='/tmp/charm/dir')
+    def test_install_hook(self, mock_add_source, mock_apt_update,
+                          mock_apt_install):
+        self.test_config.set('source', 'fake-source')
+        self.test_config.set('key', 'fake-key')
+
+        hooks.install_hook()
+        mock_add_source.assert_called_with('fake-source', 'fake-key')
+        mock_apt_update.assert_called_with(fatal=True)
+        mock_apt_install.assert_called_with(packages=hooks.INSTALL_PACKAGES,
+                                            fatal=True)
+
+    @patch.object(hooks, 'run_admin_command')
+    @patch.object(hooks, 'Connection')
+    @patch('time.sleep')
+    def test_am_i_primary(self, mock_sleep, mock_mongo_client,
+                          mock_run_admin_cmd):
+        mock_run_admin_cmd.side_effect = [{'myState': x} for x in xrange(5)]
+        expected_results = [True if x == 1 else False for x in xrange(5)]
+
+        # Check expected return values each time...
+        for exp in expected_results:
+            rv = hooks.am_i_primary()
+            self.assertEqual(exp, rv)
+
+    @patch.object(hooks, 'run_admin_command')
+    @patch.object(hooks, 'Connection')
+    @patch('time.sleep')
+    def test_am_i_primary_too_many_attempts(self, mock_sleep,
+                                            mock_mongo_client,
+                                            mock_run_admin_cmd):
+        msg = 'replSetInitiate - should come online shortly'
+        mock_run_admin_cmd.side_effect = [OperationFailure(msg)
+                                          for x in xrange(10)]
+
+        try:
+            hooks.am_i_primary()
+            self.assertTrue(False, 'Expected failure.')
+        except hooks.TimeoutException:
+            self.assertEqual(mock_run_admin_cmd.call_count, 10)
+            pass
+
+    @patch.object(hooks, 'run_admin_command')
+    @patch.object(hooks, 'Connection')
+    @patch('time.sleep')
+    def test_am_i_primary_operation_failures(self, mock_sleep,
+                                             mock_mongo_client,
+                                             mock_run_admin_cmd):
+        mock_run_admin_cmd.side_effect = OperationFailure('EMPTYCONFIG')
+
+        rv = hooks.am_i_primary()
+        mock_run_admin_cmd.assert_called()
+        self.assertFalse(rv)
+
+        mock_run_admin_cmd.reset_mock()
+        mock_run_admin_cmd.side_effect = OperationFailure('unexpected failure')
+        try:
+            hooks.am_i_primary()
+            self.assertFalse(True, "Expected OperationFailure to be raised")
+        except OperationFailure:
+            mock_run_admin_cmd.assert_called()
+
+    @patch('time.sleep')
+    @patch('subprocess.check_output')
+    def test_mongo_client_smart_no_command(self, mock_check_output,
+                                           mock_sleep):
+        rv = hooks.mongo_client_smart()
+        self.assertFalse(rv)
+        self.assertEqual(0, mock_check_output.call_count)
+
+        mock_check_output.reset_mock()
+        mock_check_output.return_value = '{"ok": 1}'
+
+        rv = hooks.mongo_client_smart(command='fake-cmd')
+        self.assertTrue(rv)
+        mock_check_output.assert_called_once_with(['mongo', '--quiet',
+                                                   '--host', 'localhost',
+                                                   '--eval',
+                                                   'printjson(fake-cmd)'])
+
+    @patch('time.sleep')
+    @patch('subprocess.check_output')
+    def test_mongo_client_smart_error_cases(self, mock_ck_output, mock_sleep):
+        mock_ck_output.side_effect = [CalledProcessError(1, 'cmd',
+                                                         output='fake-error')
+                                      for x in xrange(11)]
+        rv = hooks.mongo_client_smart(command='fake-cmd')
+        self.assertFalse(rv)
+
+    @patch('subprocess.call')
+    def test_mongo_client(self, mock_subprocess):
+        rv = hooks.mongo_client()
+        self.assertFalse(rv)
+        self.assertEqual(0, mock_subprocess.call_count)
+
+        mock_subprocess.reset_mock()
+        rv = hooks.mongo_client(host='fake-host')
+        self.assertFalse(rv)
+        self.assertEqual(0, mock_subprocess.call_count)
+
+        mock_subprocess.reset_mock()
+        rv = hooks.mongo_client(command='fake-command')
+        self.assertFalse(rv)
+        self.assertEqual(0, mock_subprocess.call_count)
+
+        mock_subprocess.reset_mock()
+        mock_subprocess.return_value = 0
+        rv = hooks.mongo_client(host='fake-host', command='fake-command')
+        expected_cmd = ("mongo --host %s --eval 'printjson(%s)'"
+                        % ('fake-host', 'fake-command'))
+        mock_subprocess.assert_called_once_with(expected_cmd, shell=True)
+        self.assertTrue(rv)
+
+        mock_subprocess.reset_mock()
+        mock_subprocess.return_value = 1
+        rv = hooks.mongo_client(host='fake-host', command='fake-command')
+        expected_cmd = ("mongo --host %s --eval 'printjson(%s)'"
+                        % ('fake-host', 'fake-command'))
+        mock_subprocess.assert_called_once_with(expected_cmd, shell=True)
+        self.assertFalse(rv)
+
+    @patch.object(hooks, 'am_i_primary')
+    @patch.object(hooks, 'init_replset')
+    @patch.object(hooks, 'relation_get')
+    @patch.object(hooks, 'peer_units')
+    @patch.object(hooks, 'oldest_peer')
+    @patch.object(hooks, 'join_replset')
+    @patch.object(hooks, 'unit_get')
+    def test_replica_set_relation_changed(self, mock_unit_get,
+                                          mock_join_replset, mock_oldest_peer,
+                                          mock_peer_units, mock_relation_get,
+                                          mock_init_replset, mock_is_primary):
+        # set the unit_get('private-address')
+        mock_unit_get.return_value = 'juju-local-unit-0.local'
+        mock_relation_get.return_value = None
+
+        # Test when remote hostname is None, should not join
+        hooks.replica_set_relation_changed()
+        self.assertEqual(0, mock_join_replset.call_count)
+
+        # Test remote hostname is valid, but master is somehow not defined
+        mock_join_replset.reset_mock()
+        mock_relation_get.return_value = 'juju-local-unit-0'
+
+        hooks.replica_set_relation_changed()
+
+        self.assertEqual(1, mock_join_replset.call_count)
+
+        # Test when not oldest peer, don't init replica set
+        mock_init_replset.reset_mock()
+        mock_oldest_peer.reset_mock()
+        mock_peer_units.return_value = ['mongodb/1', 'mongodb/2']
+        mock_oldest_peer.return_value = False
+
+        hooks.replica_set_relation_changed()
+
+        self.assertEqual(mock_init_replset.call_count, 0)
+
+        # Test when its also the PRIMARY
+        mock_relation_get.reset_mock()
+        mock_relation_get.side_effect = ['juju-remote-unit-0', '12345']
+        mock_oldest_peer.reset_mock()
+        mock_oldest_peer.return_value = False
+        mock_is_primary.reset_mock()
+        mock_is_primary.return_value = True
+
+        hooks.replica_set_relation_changed()
+        call1 = call('juju-local-unit-0.local:27017',
+                     'juju-remote-unit-0:12345')
+        mock_join_replset.assert_has_calls(call1)
+
+    @patch.object(hooks, 'unit_get')
+    @patch.object(hooks, 'leave_replset')
+    @patch.object(hooks, 'am_i_primary')
+    def test_replica_set_relation_departed(self, mock_am_i_primary,
+                                           mock_leave_replset, mock_unit_get):
+        mock_am_i_primary.return_value = False
+        hooks.replica_set_relation_departed()
+
+        self.assertEqual(0, mock_leave_replset.call_count)
+
+        mock_am_i_primary.reset_mock()
+        mock_am_i_primary.return_value = True
+        mock_unit_get.return_value = 'juju-local'
+
+        self.test_relation.set({'hostname': 'juju-remote',
+                                'port': '27017'})
+        mock_leave_replset.reset_mock()
+
+        hooks.replica_set_relation_departed()
+
+        call1 = call('juju-local:27017', 'juju-remote:27017')
+        mock_leave_replset.assert_has_calls(call1)

=== added file 'unit_tests/test_utils.py'
--- unit_tests/test_utils.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_utils.py	2015-01-22 15:32:18 +0000
@@ -0,0 +1,111 @@
+import logging
+import unittest
+import os
+import yaml
+import io
+
+from contextlib import contextmanager
+from mock import patch
+
+
+@contextmanager
+def mock_open(filename, contents=None):
+    ''' Slightly simpler mock of open to return contents for filename '''
+    def mock_file(*args):
+        if args[0] == filename:
+            return io.StringIO(contents)
+        else:
+            return open(*args)
+    with patch('__builtin__.open', mock_file):
+        yield
+
+
+def load_config():
+    '''
+    Walk backwords from __file__ looking for config.yaml, load and return the
+    'options' section'
+    '''
+    config = None
+    f = __file__
+    while config is None:
+        d = os.path.dirname(f)
+        if os.path.isfile(os.path.join(d, 'config.yaml')):
+            config = os.path.join(d, 'config.yaml')
+            break
+        f = d
+
+    if not config:
+        logging.error('Could not find config.yaml in any parent directory '
+                      'of %s. ' % file)
+        raise Exception
+
+    return yaml.safe_load(open(config).read())['options']
+
+
+def get_default_config():
+    '''
+    Load default charm config from config.yaml return as a dict.
+    If no default is set in config.yaml, its value is None.
+    '''
+    default_config = {}
+    config = load_config()
+    for k, v in config.iteritems():
+        if 'default' in v:
+            default_config[k] = v['default']
+        else:
+            default_config[k] = None
+    return default_config
+
+
+class CharmTestCase(unittest.TestCase):
+    def setUp(self, obj, patches):
+        super(CharmTestCase, self).setUp()
+        self.patches = patches
+        self.obj = obj
+        self.test_config = TestConfig()
+        self.test_relation = TestRelation()
+        self.patch_all()
+
+    def patch(self, method):
+        _m = patch.object(self.obj, method)
+        mock = _m.start()
+        self.addCleanup(_m.stop)
+        return mock
+
+    def patch_all(self):
+        for method in self.patches:
+            setattr(self, method, self.patch(method))
+
+
+class TestConfig(object):
+    def __init__(self):
+        self.config = get_default_config()
+
+    def get(self, attr):
+        try:
+            return self.config[attr]
+        except KeyError:
+            return None
+
+    def get_all(self):
+        return self.config
+
+    def set(self, attr, value):
+        if attr not in self.config:
+            raise KeyError
+        self.config[attr] = value
+
+
+class TestRelation(object):
+    def __init__(self, relation_data={}):
+        self.relation_data = relation_data
+
+    def set(self, relation_data):
+        self.relation_data = relation_data
+
+    def get(self, attr=None, unit=None, rid=None):
+        if attr is None:
+            return self.relation_data
+        elif attr in self.relation_data:
+            return self.relation_data.get(attr)
+        return None