Merge lp:~james-page/charms/precise/hadoop/storage-options into lp:~charmers/charms/precise/hadoop/trunk

Proposed by James Page
Status: Merged
Merged at revision: 27
Proposed branch: lp:~james-page/charms/precise/hadoop/storage-options
Merge into: lp:~charmers/charms/precise/hadoop/trunk
Diff against target: 80 lines (+27/-3)
3 files modified
config.yaml (+10/-0)
hooks/hadoop-common (+16/-2)
revision (+1/-1)
To merge this branch: bzr merge lp:~james-page/charms/precise/hadoop/storage-options
Reviewer Review Type Date Requested Status
Marc Cluet (community) Approve
Review via email: mp+108194@code.launchpad.net

Description of the change

This branch adds a storage option which allows all hadoop data to be stored
in a filesystem outside of root.

This supports using /mnt in ec2 (mainly)

To post a comment you must log in.
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Hi James. My only question would be whether or not we can just make it so the data is inaccessible if the value is changed, and can be made accessible again if it is changed back. "Lose all your data" sounds pretty scary.

Revision history for this message
James Page (james-page) wrote :

Clint

Guess you could just switch it back - description updated accordingly.

Revision history for this message
Marc Cluet (lynxman) wrote :

Change looks good, thank you very much James!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'config.yaml'
--- config.yaml 2012-04-24 13:23:02 +0000
+++ config.yaml 2012-06-07 13:07:25 +0000
@@ -142,3 +142,13 @@
142 description: |142 description: |
143 The number of worker threads that for the http server. This is used for143 The number of worker threads that for the http server. This is used for
144 map output fetching.144 map output fetching.
145 hadoop.dir.base:
146 type: string
147 default: /var/lib/hadoop
148 description: |
149 The directory under which all other hadoop data is stored. Use this
150 to take advantage of extra storage that might be avaliable.
151 .
152 You can change this in a running deployment but all existing data in
153 HDFS will be inaccessible; you can of course switch it back if you
154 do this by mistake.
145155
=== modified file 'hooks/hadoop-common'
--- hooks/hadoop-common 2012-03-27 12:00:13 +0000
+++ hooks/hadoop-common 2012-06-07 13:07:25 +0000
@@ -159,7 +159,7 @@
159 # Purge existing configuration159 # Purge existing configuration
160 rm -f $dir/1*-dfs.*160 rm -f $dir/1*-dfs.*
161 config_element "dfs.name.dir" \161 config_element "dfs.name.dir" \
162 "/var/lib/hadoop/cache/hadoop/dfs/name" > \162 "`config-get hadoop.dir.base`/cache/hadoop/dfs/name" > \
163 $dir/10-dfs.name.dir163 $dir/10-dfs.name.dir
164 config_element "dfs.namenode.handler.count" \164 config_element "dfs.namenode.handler.count" \
165 "`config-get dfs.namenode.handler.count`" > \165 "`config-get dfs.namenode.handler.count`" > \
@@ -196,13 +196,25 @@
196 dir=`dotdee --dir /etc/hadoop/conf.juju/core-site.xml`196 dir=`dotdee --dir /etc/hadoop/conf.juju/core-site.xml`
197 config_basic $dir197 config_basic $dir
198 rm -f $dir/1*-*198 rm -f $dir/1*-*
199 config_element "hadoop.tmp.dir" "/var/lib/hadoop/cache/\${user.name}" > \199 config_element "hadoop.tmp.dir" "`config-get hadoop.dir.base`/cache/\${user.name}" > \
200 $dir/10-hadoop.tmp.dir200 $dir/10-hadoop.tmp.dir
201 config_element "io.file.buffer.size" "`config-get io.file.buffer.size`" > \201 config_element "io.file.buffer.size" "`config-get io.file.buffer.size`" > \
202 $dir/11-io.file.buffer.size202 $dir/11-io.file.buffer.size
203 dotdee --update /etc/hadoop/conf.juju/core-site.xml || true203 dotdee --update /etc/hadoop/conf.juju/core-site.xml || true
204}204}
205205
206configure_tmp_dir_perms() {
207 dir=`config-get hadoop.dir.base`
208 # Make sure the directory exists
209 mkdir -p $dir/cache/hadoop
210 # We don't want to do this recursively since we may be reinstalling, in which case
211 # users have their own cache/<username> directories which shouldn't be stolen
212 chown root:hadoop $dir $dir/cache $dir/cache/hadoop
213 # Ensure group write on this directory or we can start namenode/datanode
214 chmod 775 $dir/cache/hadoop
215 chmod 1777 $dir/cache
216}
217
206configure_role_relation () {218configure_role_relation () {
207 dir=`dotdee --dir /etc/hadoop/conf.juju/core-site.xml`219 dir=`dotdee --dir /etc/hadoop/conf.juju/core-site.xml`
208 juju-log "Configuring service unit relation $1..."220 juju-log "Configuring service unit relation $1..."
@@ -368,6 +380,7 @@
368 install_base_packages380 install_base_packages
369 install_optional_packages381 install_optional_packages
370 configure_hadoop382 configure_hadoop
383 configure_tmp_dir_perms
371 ;;384 ;;
372 jobtracker-relation-joined)385 jobtracker-relation-joined)
373 case $mapred_role in386 case $mapred_role in
@@ -541,6 +554,7 @@
541 upgrade-charm|config-changed)554 upgrade-charm|config-changed)
542 install_optional_packages555 install_optional_packages
543 configure_hadoop556 configure_hadoop
557 configure_tmp_dir_perms
544 conditional_restart # only restart if pertinent config has changed!558 conditional_restart # only restart if pertinent config has changed!
545 open_ports559 open_ports
546 ;;560 ;;
547561
=== modified file 'revision'
--- revision 2012-04-24 13:23:02 +0000
+++ revision 2012-06-07 13:07:25 +0000
@@ -1,1 +1,1 @@
115116

Subscribers

People subscribed via source and target branches

to all changes: