Merge lp:~rharding/charms/precise/mongodb/persistent_storage into lp:charms/mongodb

[Richard Harding]

Add 'persistent' config option to store mongo data.

Adds support for using persistant storage for mongo's data and meta files.

To start with it adds a new config `persistent`. It assumes you've attached a persistent volume to the host. There's a section of docs in the readme with the command expected for adding it.

[lp:~rharding/charms/precise/mongodb/persistent_storage]

35. By Richard Harding on 2013-02-12

Remove msg levels not required

[Juan L. Negron (negronjl)]

Reviewing this now.

-Juan

[Juan L. Negron (negronjl)]

Although the idea is a really good one, I don't approve. I explain:

The persistent flag works, the logic where MongoDB tries to figure out where the newly created mount point is located, is subject to errors. In my opinion, it would be best to pass the device ( /dev/vdxx ) as a config option. That would alleviate some of the existing logic as it would not have to figure out where the device is. The rationale of my objection is that by passing the device explicitly, there is a chance of passing multiple devices and the charm would still work.

In my opinion, there should be a config options where the user could pass the device explicitly. Of course, this option would only be evaluated if the persistent config option is set to True.

Thanks,

Juan

review: Disapprove

Unmerged revisions

35. By Richard Harding on 2013-02-12

Remove msg levels not required

34. By Richard Harding on 2013-02-12

Fix the revision back to original + 1

33. By Richard Harding on 2013-02-12

Add some extra checks

32. By Richard Harding on 2013-02-12

Garden

31. By Richard Harding on 2013-02-12

Update docs

30. By Richard Harding on 2013-02-12

Update to set up a unformatted disk

29. By Richard Harding on 2013-02-11

almost have it working, unmounts after config change somehow

=== modified file 'README.md'
--- README.md	2012-11-18 07:52:57 +0000
+++ README.md	2013-02-12 17:53:21 +0000
@@ -279,7 +279,7 @@
     configsvr:
       replicaset: configsvr
 We'll save this one as ~/mongodb-shard.yaml
-  
+
 
 ### Bootstrap the environment
     juju bootstrap
@@ -303,6 +303,20 @@
     juju add-realtion mongos:mongos shard2:database
     juju add-realtion mongos:mongos shard3:database
 
+## Using persistent storage for data.
+
+The script will format a newly created persistent volume if it's not been
+formatted already.
+
+    # Create a new 30gb volume.
+    euca-attach-volume -z nova -s 30
+    euca-describe-instances
+    euca-describe-volumes
+    euca-attach-volume -d /dev/vdz -i i-00006c74 vol-000000e2
+    # Wait for the volume to attach.
+    juju set mongodb persistent=True dbpath=/mnt/mongodb
+
+
 With the above commands, we should now have a three replica-set sharded cluster running.
 Using the default configuration, here are some details of our sharded cluster:
 - mongos is running on port 27021

=== modified file 'config.yaml'
--- config.yaml	2012-08-19 02:31:14 +0000
+++ config.yaml	2013-02-12 17:53:21 +0000
@@ -1,4 +1,8 @@
 options:
+  persistent:
+    default: False
+    type: boolean
+    description: use mounted persistent storage for mongodb.
   dbpath:
     default: "/var/lib/mongodb"
     type: string

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2012-11-17 03:28:16 +0000
+++ hooks/hooks.py	2013-02-12 17:53:21 +0000
@@ -13,7 +13,10 @@
 import signal
 import socket
 import time
-
+from grp import getgrnam
+from os import chown
+from os import mkdir
+from pwd import getpwnam
 
 ###############################################################################
 # Supporting functions
@@ -845,6 +848,10 @@
     print "config_data: ", config_data
     mongodb_config = open(default_mongodb_config).read()
 
+    # Setup persistent storage if required.
+    if config_data.get('persistent') is True:
+        mount_persistent_storage()
+
     # current ports
     current_mongodb_port = re.search('^#*port\s+=\s+(\w+)',
         mongodb_config,
@@ -1126,6 +1133,72 @@
     return(retVal)
 
 
+#------------------------------------------------------------------------------
+# run: Run a command, return the output
+#------------------------------------------------------------------------------
+def run(command, exit_on_error=True):
+    try:
+        juju_log(command)
+        return subprocess.check_output(command, shell=True)
+    except subprocess.CalledProcessError, e:
+        juju_log("status=%d, output=%s" % (e.returncode, e.output))
+        if exit_on_error:
+            sys.exit(e.returncode)
+        else:
+            raise
+
+
+def mount_persistent_storage():
+    """If true, attempt to mount mongodb onto the persistent storage."""
+    juju_log("Mounting persistent storage")
+    mongo_dir = '/mnt/mongodb'
+
+    dmesg = run("dmesg | grep 'unknown partition table' | tail -n1")
+    device_id = re.search('vd.:', dmesg)
+    if device_id is None:
+        juju_log('no device found in dmesg output: ' + dmesg)
+    else:
+        device_id = device_id.group().rstrip(':')
+
+        device_path = '/dev/' + device_id
+        juju_log('mounting device: ' + device_path)
+
+        # check for file system on the device to mount.
+        device_info = run('file -s ' + device_path)
+
+        # Make sure we don't run this on some unsuspecting ext3 device on
+        # the machine.
+        if not 'ext4' in device_info and 'ext3' in device_info:
+            # sudo mke2fs -L dataset-1 -m 0 -T ext4 /dev/vdc
+            juju_log("Existing filesystem on %s not found. %s" % (
+                device_path,
+                device_info))
+            juju_log("Making filesystem on device: %s" % device_path)
+            format_cmd = "mke2fs -L dataset-1 -m 0 -T ext4 %s" % device_path
+            run(format_cmd)
+
+        # mount /dev/vde /mnt
+        res = run('mount')
+        if device_path not in res:
+            juju_log("Mounting device to /mnt")
+            command = "mount %s /mnt" % device_path
+            res = run(command)
+        else:
+            juju_log("Mount point already found")
+            juju_log(res)
+
+        # mkdir /mnt/mongodb
+        if not os.path.isdir(mongo_dir):
+            mkdir(mongo_dir)
+
+        # chown -R mongodb:mongodb /mnt/mongodb
+        chown(
+            mongo_dir,
+            getpwnam('mongodb').pw_uid,
+            getgrnam('mongodb').gr_gid
+        )
+
+
 def mongos_relation_broken():
 #    config_servers = load_config_servers(default_mongos_list)
 #    for member in relation_list():

=== modified file 'revision'
--- revision	2012-11-18 07:54:22 +0000
+++ revision	2013-02-12 17:53:21 +0000
@@ -1,1 +1,1 @@
-25
+29