Merge lp:~billy-olsen/charms/trusty/mongodb/lp1513094 into lp:charms/trusty/mongodb

[Billy Olsen]

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #2558 mongodb for billy-olsen mp294565
    LINT OK: passed

Build: http://10.245.162.36:8080/job/charm_lint_check/2558/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #1989 mongodb for billy-olsen mp294565
    UNIT OK: passed

Build: http://10.245.162.36:8080/job/charm_unit_test/1989/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #743 mongodb for billy-olsen mp294565
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [functional_test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/16381918/
Build: http://10.245.162.36:8080/job/charm_amulet_test/743/

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/4174/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/4187/

review: Needs Fixing (automated testing)

[Andrew McLeod (admcleod)]

Hi there Billy,

This is failing with the following. I wonder if this is just a matter of naming in the tests?

bundletester -vFt tests/ -l DEBUG -e sneerAWS --no-destroy

With the following 2 errors:

Failed to find running MongoDB on host config-sentry
Failed to find running MongoDB on host mongodb1-sentry

review: Needs Fixing

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #6 mongodb for billy-olsen mp294565
    UNIT FAIL: unit-test failed

UNIT Results (max last 2 lines):
Makefile:35: recipe for target 'test' failed
ERROR:root:Make target returned non-zero.

Full unit test output: http://paste.ubuntu.com/23110021/
Build: http://10.245.162.208:8080/job/charm_unit_test/6/

[Ryan Beisner (1chb1n)]

Can we please refactor the unit test and bundletester test environments here to build a virtualenv, use tox, ostestr, etc., a la OpenStack Charms? Extra credit for that! ;-)

Bottom line is the test suite won't pass anymore in OSCI, as we're getting very strict with tests that try to bring in arbitrary debs.

The litmus test is this:

A fresh Xenial instance. Juju, juju-deployer, charm-tools debs installed. Everything else in a venv. Tests should exec and pass.

Thank you!

review: Needs Information

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #17 mongodb for billy-olsen mp294565
    LINT OK: passed

Build: http://10.245.162.208:8080/job/charm_lint_check/17/

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2016-02-26 14:00:17 +0000
+++ hooks/hooks.py	2016-05-12 20:29:50 +0000
@@ -46,6 +46,7 @@
 from charmhelpers.core.hookenv import (
     close_port,
     config,
+    is_relation_made,
     open_port,
     unit_get,
     relation_get,
@@ -331,6 +332,10 @@
     if not config_data['web_admin_ui']:
         config.append("nohttpinterface = true")
         config.append("")
+    else:
+        config.append("rest = true")
+        config.append("httpinterface = true")
+        config.append("")
 
     # noscripting
     if config_data['noscripting']:
@@ -367,14 +372,23 @@
         config.append("mms-interval = %s" % config_data['mms-interval'])
         config.append("")
 
-    # master/slave
-    if config_data['master'] == "self":
-        config.append("master = true")
+    # Set either replica-set or master, depending upon whether the
+    # the replica-set (peer) relation is established. If the user
+    # chooses to use juju scale out (e.g. juju add-unit) then the
+    # charm will use replica-set replication. The user may opt to
+    # do master/slave replication or sharding as a different form
+    # of scaleout.
+    if is_relation_made('replica-set'):
+        config.append("replSet = %s" % config_data['replicaset'])
         config.append("")
     else:
-        config.append("slave = true")
-        config.append("source = %s" % config_data['master'])
-        config.append("")
+        if config_data['master'] == "self":
+            config.append("master = true")
+            config.append("")
+        else:
+            config.append("slave = true")
+            config.append("source = %s" % config_data['master'])
+            config.append("")
 
     # arbiter
     if config_data['arbiter'] != "disabled" and \
@@ -405,6 +419,27 @@
     return('\n'.join(config))
 
 
+def get_current_mongo_config():
+    """Reads the current mongo configuration file and returns a dict
+    containing the key/value pairs found in the configuration file.
+
+    :returns dict: key/value pairs of the configuration file.
+    """
+    results = {}
+    with open(default_mongodb_config, 'r') as f:
+        for line in f:
+            line = line.strip()
+
+            # Skip over comments, blank lines, and any other line
+            # that appears to not contain a key = value pair.
+            if line.startswith('#') or '=' not in line:
+                continue
+
+            key, value = line.split('=', 1)
+            results[key.strip()] = value.strip()
+    return results
+
+
 def mongo_client(host=None, command=None):
     if host is None or command is None:
         return(False)
@@ -541,22 +576,19 @@
     try:
         juju_log('enable_replset: trying to get lock on: %s' %
                  default_mongodb_init_config)
-        with FileLock(INIT_LOCKFILE):
-            juju_log('enable_replset: lock acquired', level=DEBUG)
-            with open(default_mongodb_init_config) as mongo_init_file:
-                mongodb_init_config = mongo_init_file.read()
-            if re.search(' --replSet %s ' % replicaset_name,
-                         mongodb_init_config, re.MULTILINE) is None:
-                juju_log('enable_replset: --replset not preset,'
-                         ' enabling',
-                         level=DEBUG)
-                mongodb_init_config = regex_sub([(' -- ',
-                                                  ' -- --replSet %s ' %
-                                                  replicaset_name)],
-                                                mongodb_init_config)
-                update_file(default_mongodb_init_config,
-                            mongodb_init_config)
-                retVal = True
+
+        current_config = get_current_mongo_config()
+        config_data = config()
+
+        if 'replSet' in current_config and \
+                current_config['replSet'] == config_data['replicaset']:
+            juju_log('enable_replset: replica set is already enabled',
+                     level=DEBUG)
+        else:
+            juju_log('enable_replset: enabling replicaset %s' %
+                     config_data['replicaset'], level=DEBUG)
+            mongodb_config = mongodb_conf(config_data)
+            retVal = update_file(default_mongodb_config, mongodb_config)
 
         juju_log('enable_replset will return: %s' % str(retVal), level=DEBUG)
 
@@ -582,64 +614,6 @@
     return(update_file(default_mongodb_init_config, mongodb_init_config))
 
 
-def disable_replset(replicaset_name=None):
-    if replicaset_name is None:
-        retVal = False
-    try:
-        mongodb_init_config = open(default_mongodb_init_config).read()
-        if re.search(' --replSet %s ' % replicaset_name,
-                     mongodb_init_config, re.MULTILINE) is not None:
-            mongodb_init_config = regex_sub([
-                (' --replSet %s ' % replicaset_name, ' ')
-                ], mongodb_init_config)
-        retVal = update_file(default_mongodb_init_config, mongodb_init_config)
-    except Exception, e:
-        juju_log(str(e))
-        retVal = False
-    finally:
-        return(retVal)
-
-
-def enable_web_admin_ui(port=None):
-    if port is None:
-        juju_log("enable_web_admin_ui: port not defined.")
-        return(False)
-    try:
-        mongodb_init_config = open(default_mongodb_init_config).read()
-        if re.search(' --rest ', mongodb_init_config, re.MULTILINE) is None:
-            mongodb_init_config = regex_sub([(' -- ', ' -- --rest ')],
-                                            mongodb_init_config)
-        retVal = update_file(default_mongodb_init_config, mongodb_init_config)
-    except Exception, e:
-        juju_log(str(e))
-        retVal = False
-    finally:
-        if retVal:
-            open_port(port)
-        return(retVal)
-
-
-def disable_web_admin_ui(port=None):
-    if port is None:
-        juju_log("disable_web_admin_ui: port not defined.")
-        return(False)
-    try:
-        mongodb_init_config = open(default_mongodb_init_config).read()
-        if re.search(' --rest ',
-                     mongodb_init_config,
-                     re.MULTILINE) is not None:
-            mongodb_init_config = regex_sub([(' --rest ', ' ')],
-                                            mongodb_init_config)
-        retVal = update_file(default_mongodb_init_config, mongodb_init_config)
-    except Exception, e:
-        juju_log(str(e))
-        retVal = False
-    finally:
-        if retVal:
-            close_port(port)
-        return(retVal)
-
-
 def enable_arbiter(master_node=None, host=None):
     juju_log("enable_arbiter: master_node: %s, host: %s" %
              (master_node, host))
@@ -1002,9 +976,9 @@
 
     # web_admin_ui
     if config_data['web_admin_ui']:
-        enable_web_admin_ui(new_web_admin_ui_port)
+        open_port(new_web_admin_ui_port)
     else:
-        disable_web_admin_ui(current_web_admin_ui_port)
+        close_port(current_web_admin_ui_port)
 
     # replicaset_master
     if config_data['replicaset_master'] != "auto":
@@ -1141,7 +1115,7 @@
     relation_set(relation_id(), relation_data)
 
 
[e-mail]('replicaset-relation-joined')
[e-mail]('replica-set-relation-joined')
 def replica_set_relation_joined():
     juju_log("replica_set_relation_joined-start")
     my_hostname = unit_get('private-address')
@@ -1194,7 +1168,7 @@
     raise TimeoutException('Unable to determine if local unit is primary')
 
 
[e-mail]('replicaset-relation-changed')
[e-mail]('replica-set-relation-changed')
 def replica_set_relation_changed():
     private_address = unit_get('private-address')
     remote_hostname = relation_get('hostname')
@@ -1228,7 +1202,7 @@
     juju_log('replica_set_relation_changed-finish')
 
 
[e-mail]('replicaset-relation-departed')
[e-mail]('replica-set-relation-departed')
 def replica_set_relation_departed():
     juju_log('replica_set_relation_departed-start')
 
@@ -1259,7 +1233,7 @@
     juju_log('replica_set_relation_departed-finish')
 
 
[e-mail]('replicaset-relation-broken')
[e-mail]('replica-set-relation-broken')
 def replica_set_relation_broken():
     juju_log('replica_set_relation_broken-start')
 

=== modified file 'tests/01_deploy_single.py'
--- tests/01_deploy_single.py	2015-11-17 22:00:56 +0000
+++ tests/01_deploy_single.py	2016-05-12 20:29:50 +0000
@@ -36,7 +36,7 @@
         amulet.raise_status(amulet.FAIL, msg="Failed to insert test data")
     # Can we delete from a shard using the Mongos hub?
     result = db.amulet.remove(insert_id)
-    if result['err'] is not None:
+    if 'err' in result and result['err'] is not None:
         amulet.raise_status(amulet.FAIL, msg="Failed to remove test data")
 
 

=== modified file 'tests/02_deploy_shard_test.py'
--- tests/02_deploy_shard_test.py	2015-12-03 22:49:56 +0000
+++ tests/02_deploy_shard_test.py	2016-05-12 20:29:50 +0000
@@ -92,7 +92,7 @@
 def validate_running_services():
     for service in sentry_dict:
         output = sentry_dict[service].run('service mongodb status')
-        service_active = str(output).find('mongodb start/running')
+        service_active = str(output).find('active (running)')
         if service_active == -1:
             message = "Failed to find running MongoDB on host {}".format(
                 service)
@@ -115,7 +115,7 @@
         amulet.raise_status(amulet.FAIL, msg="Failed to insert test data")
     # Can we delete from a shard using the Mongos hub?
     result = db.amulet.remove(insert_id)
-    if result['err'] is not None:
+    if 'err' in result and result['err'] is not None:
         amulet.raise_status(amulet.FAIL, msg="Failed to remove test data")
 
 

=== modified file 'tests/03_deploy_replicaset.py'
--- tests/03_deploy_replicaset.py	2015-12-02 22:50:09 +0000
+++ tests/03_deploy_replicaset.py	2016-05-12 20:29:50 +0000
@@ -63,7 +63,7 @@
 def validate_running_services():
     for service in sentry_dict:
         output = sentry_dict[service].run('service mongodb status')
-        service_active = str(output).find('mongodb start/running')
+        service_active = str(output).find('active (running)')
         if service_active == -1:
             message = "Failed to find running MongoDB on host {}".format(
                 service)
@@ -158,7 +158,7 @@
         amulet.raise_status(amulet.FAIL, msg="Failed to insert test data")
     # Can we delete from a shard using the Mongos hub?
     result = db.amulet.remove(insert_id)
-    if result['err'] is not None:
+    if 'err' in result and result['err'] is not None:
         amulet.raise_status(amulet.FAIL, msg="Failed to remove test data")
 
 

=== modified file 'unit_tests/test_hooks.py'
--- unit_tests/test_hooks.py	2015-07-28 18:53:19 +0000
+++ unit_tests/test_hooks.py	2016-05-12 20:29:50 +0000
@@ -3,6 +3,7 @@
 import hooks
 
 from test_utils import CharmTestCase
+from test_utils import mock_open
 from pymongo.errors import OperationFailure
 from subprocess import CalledProcessError
 
@@ -339,3 +340,20 @@
 
         call1 = call('juju-local:27017', 'juju-remote:27017')
         mock_leave_replset.assert_has_calls([call1])
+
+    def test_get_current_mongo_config(self):
+        test_config = u"""
+        # Comment
+        key = value
+        one=two
+
+        three =four
+        """
+        expected = {
+            'key': 'value',
+            'one': 'two',
+            'three': 'four'
+        }
+        with mock_open('/etc/mongodb.conf', test_config):
+            results = hooks.get_current_mongo_config()
+            self.assertEqual(results, expected)