Merge lp:~hazmat/charms/precise/hadoop/trunk into lp:~charmers/charms/precise/hadoop/trunk

[Gary Poster]

Switch config values from dots to dashes. This makes the charm work with Juju Core and the GUI.

This branch from Kapil includes Jeff Pihach's fix for the hadoop charm.

The hadoop charm uses dots in the config values. This breaks the CLI (juju set will fail) and the GUI (deployment will fail). It may be a conscious decision in the CLI to reject config values with dots. Whether or not that is so, as a simple practical solution, we are declaring dots as inappropriate for config values, until/unless changing this is a high enough priority for Juju Core and the GUI.

The fix here is to convert dots to hyphens.

Thanks!

Gary

[Marco Ceppi (marcoceppi)]

I'm strongly opposed to changes that break backwards compatibility. However, I'm letting this one go through, since you'd need juju < 0.7 to successfully deploy and change config for this charm - given that these releases are all depreciated it's recommended people move forward. Since you can't change config options for this charm in juju-core anyone upgrading won't be affected.

That being said, I'd like to state that all updates to charms must maintain backwards compatibility. Shortcomings, bugs, and differences between "pyjuju" and "gojuju" shouldn't be addressed by changing (and breaking) charms.

review: Approve

=== modified file 'config.yaml'
--- config.yaml	2012-06-07 13:03:51 +0000
+++ config.yaml	2013-10-15 19:16:27 +0000
@@ -66,29 +66,29 @@
       .
       If you are also mixing MapReduce and DFS roles on the same units you need to
       take this into account as well (see README for more details).
-# Expert options See http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F 
+# Expert options See http://wiki.apache.org/hadoop/FAQ#How_well_does_Hadoop_scale.3F
 # for more details
-  dfs.namenode.handler.count:
+  dfs_namenode_handler_count:
     type: int
     default: 10
     description: |
       The number of server threads for the namenode.  Increase this in larger
       deployments to ensure the namenode can cope with the number of datanodes
       that it has to deal with.
-  dfs.block.size:
+  dfs_block_size:
     type: int
     default: 67108864
     description: |
-      The default block size for new files (default to 64MB).  Increase this in 
+      The default block size for new files (default to 64MB).  Increase this in
       larger deployments for better large data set performance.
-  io.file.buffer.size:
+  io_file_buffer_size:
     type: int
     default: 4096
     description: |
       The size of buffer for use in sequence files. The size of this buffer should
       probably be a multiple of hardware page size (4096 on Intel x86), and it
       determines how much data is buffered during read and write operations.
-  dfs.datanode.max.xcievers:
+  dfs_datanode_max_xcievers:
     type: int
     default: 4096
     description: |
@@ -97,13 +97,13 @@
       An Hadoop HDFS datanode has an upper bound on the number of files that it
       will serve at any one time. This defaults to 256 (which is low) in hadoop
       1.x - however this charm increases that to 4096.
-  mapred.reduce.parallel.copies:
+  mapred_reduce_parallel_copies:
     type: int
     default: 5
     description: |
       The default number of parallel transfers run by reduce during the
       copy(shuffle) phase.
-  mapred.child.java.opts:
+  mapred_child_java_opts:
     type: string
     default: -Xmx200m
     description: |
@@ -117,32 +117,32 @@
       .
       The configuration variable mapred.child.ulimit can be used to control
       the maximum virtual memory of the child processes.
-  io.sort.factor:
+  io_sort_factor:
     type: int
     default: 10
     description: |
       The number of streams to merge at once while sorting files. This
       determines the number of open file handles.
-  io.sort.mb:
+  io_sort_mb:
     type: int
     default: 100
     description: |
       The total amount of buffer memory to use while sorting files, in
       megabytes. By default, gives each merge stream 1MB, which should minimize
       seeks.
-  mapred.job.tracker.handler.count:
+  mapred_job_tracker_handler_count:
     type: int
     default: 10
     description: |
       The number of server threads for the JobTracker. This should be roughly
       4% of the number of tasktracker nodes.
-  tasktracker.http.threads:
+  tasktracker_http_threads:
     type: int
     default: 40
     description: |
       The number of worker threads that for the http server. This is used for
       map output fetching.
-  hadoop.dir.base:
+  hadoop_dir_base:
     type: string
     default: /var/lib/hadoop
     description: |

=== modified file 'hooks/hadoop-common'
--- hooks/hadoop-common	2012-05-03 22:12:08 +0000
+++ hooks/hadoop-common	2013-10-15 19:16:27 +0000
@@ -4,7 +4,7 @@
 
 configure_hosts () {
     private_address=`unit-get private-address`
-    # This is a horrible hack to ensure that 
+    # This is a horrible hack to ensure that
     # Java can resolve the hostname of the server to its
     # real IP address.
 
@@ -76,7 +76,7 @@
     case $hdfs_role in
         namenode)
             open-port 8020
-            open-port 50070 
+            open-port 50070
             ;;
         datanode)
             open-port 50010
@@ -98,13 +98,12 @@
     esac
 }
 
-MAPRED_CONFIG="mapred.reduce.parallel.copies
-mapred.child.java.opts
-io.sort.factor
-io.sort.mb
-mapred.job.tracker.handler.count
-tasktracker.http.threads
-"
+MAPRED_CONFIG="mapred_reduce_parallel_copies
+mapred_child_java_opts
+io_sort_factor
+io_sort_mb
+mapred_job_tracker_handler_count
+tasktracker_http_threads"
 
 CONFIG_FILES="/etc/hadoop/conf.juju/hdfs-site.xml
 /etc/hadoop/conf.juju/core-site.xml
@@ -143,9 +142,9 @@
         cp /dev/null /etc/hadoop/conf.juju/hdfs-site.xml
         cp /dev/null /etc/hadoop/conf.juju/core-site.xml
         cp /dev/null /etc/hadoop/conf.juju/mapred-site.xml
-        dotdee --setup /etc/hadoop/conf.juju/hdfs-site.xml 
+        dotdee --setup /etc/hadoop/conf.juju/hdfs-site.xml
         dotdee --setup /etc/hadoop/conf.juju/core-site.xml
-        dotdee --setup /etc/hadoop/conf.juju/mapred-site.xml 
+        dotdee --setup /etc/hadoop/conf.juju/mapred-site.xml
         dotdee --setup /etc/hadoop/conf.juju/hadoop-env.sh
     fi
     # Configure Heap Size
@@ -159,16 +158,16 @@
     # Purge existing configuration
     rm -f $dir/1*-dfs.*
     config_element "dfs.name.dir" \
-        "`config-get hadoop.dir.base`/cache/hadoop/dfs/name" > \
+        "`config-get hadoop_dir_base`/cache/hadoop/dfs/name" > \
         $dir/10-dfs.name.dir
     config_element "dfs.namenode.handler.count" \
-        "`config-get dfs.namenode.handler.count`" > \
+        "`config-get dfs_namenode_handler_count`" > \
         $dir/11-dfs.namenode.handler.count
     config_element "dfs.block.size" i\
-        "`config-get dfs.block.size`" > \
+        "`config-get dfs_block_size`" > \
         $dir/12-dfs.block.size
     config_element "dfs.datanode.max.xcievers" \
-        "`config-get dfs.datanode.max.xcievers`" > \
+        "`config-get dfs_datanode_max_xcievers`" > \
         $dir/13-dfs.datanode.max.xcievers
     [ "`config-get hbase`" = "True" ] && \
         config_element "dfs.support.append" "true" > \
@@ -187,7 +186,7 @@
     counter=10
     for element in $MAPRED_CONFIG
     do
-        config_element "$element" "`config-get $element`" > \
+        config_element "${element//_/.}" "`config-get $element`" > \
             $dir/20-$counter-$element
         counter=`expr $counter + 1`
     done
@@ -196,15 +195,15 @@
     dir=`dotdee --dir /etc/hadoop/conf.juju/core-site.xml`
     config_basic $dir
     rm -f $dir/1*-*
-    config_element "hadoop.tmp.dir" "`config-get hadoop.dir.base`/cache/\${user.name}" > \
+    config_element "hadoop.tmp.dir" "`config-get hadoop_dir_base`/cache/\${user.name}" > \
         $dir/10-hadoop.tmp.dir
-    config_element "io.file.buffer.size" "`config-get io.file.buffer.size`" > \
+    config_element "io.file.buffer.size" "`config-get io_file_buffer_size`" > \
         $dir/11-io.file.buffer.size
     dotdee --update /etc/hadoop/conf.juju/core-site.xml || true
 }
 
 configure_tmp_dir_perms() {
-    dir=`config-get hadoop.dir.base`
+    dir=`config-get hadoop_dir_base`
     # Make sure the directory exists
     mkdir -p $dir/cache/hadoop
     # We don't want to do this recursively since we may be reinstalling, in which case
@@ -261,7 +260,7 @@
 
 install_packages () {
     case $1 in
-        namenode|datanode|secondarynamenode|jobtracker|tasktracker) 
+        namenode|datanode|secondarynamenode|jobtracker|tasktracker)
             juju-log "Installing extra packages for $1"
             apt-get -y install hadoop-$1
             ;;
@@ -295,19 +294,19 @@
 }
 
 # Hadoop Service Control Commands
-restart_hadoop () { 
+restart_hadoop () {
     [ "$hdfs_role" != "unconfigured" ] && \
         _restart_ hadoop-$hdfs_role || :
     [ "$mapred_role" != "unconfigured" ] && \
         _restart_ hadoop-$mapred_role || :
 }
-stop_hadoop () { 
+stop_hadoop () {
     [ "$hdfs_role" != "unconfigured" ] && \
         _stop_ hadoop-$hdfs_role || :
     [ "$mapred_role" != "unconfigured" ] && \
         _stop_ hadoop-$mapred_role || :
 }
-start_hadoop () { 
+start_hadoop () {
     [ "$hdfs_role" != "unconfigured" ] && \
         _start_ hadoop-$hdfs_role || :
     [ "$mapred_role" != "unconfigured" ] && \
@@ -380,7 +379,7 @@
         install_base_packages
         install_optional_packages
         configure_hadoop
-        configure_tmp_dir_perms        
+        configure_tmp_dir_perms
         ;;
     jobtracker-relation-joined)
         case $mapred_role in
@@ -454,7 +453,7 @@
         ;;
     namenode-relation-joined)
         case $hdfs_role in
-            unconfigured)             
+            unconfigured)
                 juju-log "Configuring this unit as a namenode"
                 hdfs_role="namenode"
                 configure_role_relation $hdfs_role
@@ -513,7 +512,7 @@
         ;;
    datanode-relation-changed)
         case $hdfs_role in
-            unconfigured) 
+            unconfigured)
                 ready=`relation-get ready`
                 if [ -z "$ready" ]
                 then