Merge lp:~asanjar/charms/trusty/hadoop/hadoop-elk into lp:charms/hadoop
- Trusty Tahr (14.04)
- hadoop-elk
- Merge into trunk
Proposed by
amir sanjar
Status: | Merged | ||||
---|---|---|---|---|---|
Merged at revision: | 4 | ||||
Proposed branch: | lp:~asanjar/charms/trusty/hadoop/hadoop-elk | ||||
Merge into: | lp:charms/hadoop | ||||
Diff against target: |
537 lines (+151/-90) 5 files modified
README.md (+10/-0) config.yaml (+14/-1) files/upstart/defaults (+11/-11) hooks/hadoop-common (+114/-78) metadata.yaml (+2/-0) |
||||
To merge this branch: | bzr merge lp:~asanjar/charms/trusty/hadoop/hadoop-elk | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Charles Butler (community) | Approve | ||
charmers | Pending | ||
Review via email: mp+224143@code.launchpad.net |
Commit message
Description of the change
adding elasticsearch support to hadoop
To post a comment you must log in.
- 6. By amir sanjar
-
shutdown & elasticsearch changes
Preview Diff
[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1 | === modified file 'README.md' | |||
2 | --- README.md 2014-06-16 17:12:24 +0000 | |||
3 | +++ README.md 2014-07-07 14:22:54 +0000 | |||
4 | @@ -63,6 +63,7 @@ | |||
5 | 63 | juju add-relation hadoop-master:namenode hadoop-slavecluster:datanode | 63 | juju add-relation hadoop-master:namenode hadoop-slavecluster:datanode |
6 | 64 | juju add-relation hadoop-master:resourcemanager hadoop-slavecluster:nodemanager | 64 | juju add-relation hadoop-master:resourcemanager hadoop-slavecluster:nodemanager |
7 | 65 | 65 | ||
8 | 66 | |||
9 | 66 | ### Scale Out Usage: Separate HDFS and MapReduce | 67 | ### Scale Out Usage: Separate HDFS and MapReduce |
10 | 67 | 68 | ||
11 | 68 | In this configuration the HDFS and YARN deployments operate on | 69 | In this configuration the HDFS and YARN deployments operate on |
12 | @@ -85,6 +86,15 @@ | |||
13 | 85 | to be deployed onto machines with more processing power and hdfs services | 86 | to be deployed onto machines with more processing power and hdfs services |
14 | 86 | to be deployed onto machines with larger storage. | 87 | to be deployed onto machines with larger storage. |
15 | 87 | 88 | ||
16 | 89 | ### TO deploy a Hadoop service with elasticsearch service:: | ||
17 | 90 | # deploy ElasticSearch locally: | ||
18 | 91 | juju deploy elasticsearch elasticsearch | ||
19 | 92 | # elasticsearch-hadoop.jar file will be added to LIBJARS path | ||
20 | 93 | # Recommanded to use hadoop -libjars option to included elk jar file | ||
21 | 94 | juju add-unit -n elasticsearch | ||
22 | 95 | # deploy hive service by any senarios mentioned above | ||
23 | 96 | # associate Hive with elasticsearch | ||
24 | 97 | juju add-relation hadoop-master:elasticsearch elasticsearch:client | ||
25 | 88 | ## Known Limitations and Issues | 98 | ## Known Limitations and Issues |
26 | 89 | 99 | ||
27 | 90 | Note that removing the relation between namenode and datanode is destructive! | 100 | Note that removing the relation between namenode and datanode is destructive! |
28 | 91 | 101 | ||
29 | === modified file 'config.yaml' | |||
30 | --- config.yaml 2014-05-13 19:22:12 +0000 | |||
31 | +++ config.yaml 2014-07-07 14:22:54 +0000 | |||
32 | @@ -152,7 +152,7 @@ | |||
33 | 152 | map output fetching. | 152 | map output fetching. |
34 | 153 | hadoop_dir_base: | 153 | hadoop_dir_base: |
35 | 154 | type: string | 154 | type: string |
37 | 155 | default: /home/ubuntu/hadoop/data | 155 | default: /usr/local/hadoop/data |
38 | 156 | description: | | 156 | description: | |
39 | 157 | The directory under which all other hadoop data is stored. Use this | 157 | The directory under which all other hadoop data is stored. Use this |
40 | 158 | to take advantage of extra storage that might be avaliable. | 158 | to take advantage of extra storage that might be avaliable. |
41 | @@ -170,3 +170,16 @@ | |||
42 | 170 | default: org.apache.hadoop.mapred.ShuffleHandler | 170 | default: org.apache.hadoop.mapred.ShuffleHandler |
43 | 171 | description: | | 171 | description: | |
44 | 172 | Shuffle service that needs to be set for Map Reduce applications. | 172 | Shuffle service that needs to be set for Map Reduce applications. |
45 | 173 | dfs_heartbeat_interval: | ||
46 | 174 | type: int | ||
47 | 175 | default: 3 | ||
48 | 176 | description: | | ||
49 | 177 | Determines datanode heartbeat interval in seconds. | ||
50 | 178 | dfs_namenode_heartbeat_recheck_interval: | ||
51 | 179 | type: int | ||
52 | 180 | default: 300000 | ||
53 | 181 | description: | | ||
54 | 182 | Determines datanode recheck heartbeat interval in milliseconds | ||
55 | 183 | It is used to calculate the final tineout value for namenode. Calcultion process is | ||
56 | 184 | as follow: 10.30 minutes = 2 x (dfs.namenode.heartbeat.recheck-interval=5*60*1000) | ||
57 | 185 | + 10 * 1000 * (dfs.heartbeat.interval=3) | ||
58 | 173 | 186 | ||
59 | === modified file 'files/upstart/defaults' | |||
60 | --- files/upstart/defaults 2014-05-13 19:22:12 +0000 | |||
61 | +++ files/upstart/defaults 2014-07-07 14:22:54 +0000 | |||
62 | @@ -1,17 +1,17 @@ | |||
63 | 1 | #!/bin/bash | 1 | #!/bin/bash |
64 | 2 | 2 | ||
65 | 3 | 3 | ||
77 | 4 | echo "export JAVA_HOME=$1" > /etc/profile.d/hadoopenv.sh | 4 | #echo "export JAVA_HOME=$1" > /etc/profile.d/hadoopenv.sh |
78 | 5 | echo "export HADOOP_INSTALL=$2" >> /etc/profile.d/hadoopenv.sh | 5 | echo "export HADOOP_INSTALL=$1" >> /etc/profile.d/hadoopenv.sh |
79 | 6 | echo "export HADOOP_HOME=$2" >> /etc/profile.d/hadoopenv.sh | 6 | echo "export HADOOP_HOME=$1" >> /etc/profile.d/hadoopenv.sh |
80 | 7 | echo "export HADOOP_COMMON_HOME=$2" >> /etc/profile.d/hadoopenv.sh | 7 | echo "export HADOOP_COMMON_HOME=$1" >> /etc/profile.d/hadoopenv.sh |
81 | 8 | echo "export HADOOP_HDFS_HOME=$2" >> /etc/profile.d/hadoopenv.sh | 8 | echo "export HADOOP_HDFS_HOME=$1" >> /etc/profile.d/hadoopenv.sh |
82 | 9 | echo "export HADOOP_MAPRED_HOME=$2" >> /etc/profile.d/hadoopenv.sh | 9 | echo "export HADOOP_MAPRED_HOME=$1" >> /etc/profile.d/hadoopenv.sh |
83 | 10 | echo "export HADOOP_YARN_HOME=$2" >> /etc/profile.d/hadoopenv.sh | 10 | echo "export HADOOP_YARN_HOME=$1" >> /etc/profile.d/hadoopenv.sh |
84 | 11 | echo "export PATH=$1/bin:$PATH:$2/bin:$2/sbin" >> /etc/profile.d/hadoopenv.sh | 11 | echo "export PATH=$PATH:$1/bin:$1/sbin" >> /etc/profile.d/hadoopenv.sh |
85 | 12 | echo "export YARN_HOME=$2" >> /etc/profile.d/hadoopenv.sh | 12 | echo "export YARN_HOME=$1" >> /etc/profile.d/hadoopenv.sh |
86 | 13 | echo "export HADOOP_CONF_DIR=$4" >> /etc/profile.d/hadoopenv.sh | 13 | echo "export HADOOP_CONF_DIR=$2" >> /etc/profile.d/hadoopenv.sh |
87 | 14 | echo "export YARN_CONF_DIR=$4" >> /etc/profile.d/hadoopenv.sh | 14 | echo "export YARN_CONF_DIR=$2" >> /etc/profile.d/hadoopenv.sh |
88 | 15 | chmod +x /etc/profile.d/hadoopenv.sh | 15 | chmod +x /etc/profile.d/hadoopenv.sh |
89 | 16 | 16 | ||
90 | 17 | 17 | ||
91 | 18 | 18 | ||
92 | === modified file 'hooks/hadoop-common' | |||
93 | --- hooks/hadoop-common 2014-05-13 19:22:12 +0000 | |||
94 | +++ hooks/hadoop-common 2014-07-07 14:22:54 +0000 | |||
95 | @@ -19,31 +19,32 @@ | |||
96 | 19 | sed -i -e "s/^127.0.1.1\(.*$hostname.*\)/$private_address\1/" /etc/hosts | 19 | sed -i -e "s/^127.0.1.1\(.*$hostname.*\)/$private_address\1/" /etc/hosts |
97 | 20 | 20 | ||
98 | 21 | } | 21 | } |
99 | 22 | JAVA_VENDOR="openjdk" | ||
100 | 23 | JAVA_VERSION="7" | ||
101 | 24 | HADOOP_VERSION="hadoop-2.2.0" | ||
102 | 25 | PLATFORM_ARCH="amd64" | ||
103 | 26 | HADOOP_DIR="/usr/local/hadoop" | ||
104 | 27 | HADOOP_CONF_DIR="/etc/hadoop/conf.juju" | ||
105 | 28 | HADOOP_INSTALLED=$HADOOP_DIR/$HADOOP_VERSION | ||
106 | 29 | HADOOP_TMP_DIR=$HADOOP_DIR/tmp | ||
107 | 22 | 30 | ||
108 | 23 | set_hadoopenv () { | ||
109 | 24 | JAVA_VENDOR="openjdk" | ||
110 | 25 | JAVA_VERSION="7" | ||
111 | 26 | HADOOP_VERSION="hadoop-2.2.0" | ||
112 | 27 | PLATFORM_ARCH="amd64" | ||
113 | 28 | HOME_DIR="/home/ubuntu" | ||
114 | 29 | HADOOP_DIR="/home/ubuntu/hadoop" | ||
115 | 30 | HADOOP_CONF_DIR="/etc/hadoop/conf.juju" | ||
116 | 31 | HADOOP_INSTALLED=$HADOOP_DIR/$HADOOP_VERSION | ||
117 | 32 | HADOOP_TMP_DIR=$HADOOP_DIR/tmp | ||
118 | 33 | } | ||
119 | 34 | 31 | ||
120 | 35 | configure_systemenv(){ | 32 | configure_systemenv(){ |
121 | 36 | juju-log "Configuring hadoop system environment" | 33 | juju-log "Configuring hadoop system environment" |
123 | 37 | JAVA_HOME_PATH=$(sudo find /usr/ -name java-7-openjdk-*) | 34 | #JAVA_HOME_PATH=$(sudo find /usr/ -name java-7-openjdk-*) |
124 | 38 | if [ ! -f /etc/default/hadoop ] | 35 | if [ ! -f /etc/default/hadoop ] |
125 | 39 | then | 36 | then |
126 | 40 | juju-log "Configuring hadoop system environment - setting up hadoopenv.sh " | 37 | juju-log "Configuring hadoop system environment - setting up hadoopenv.sh " |
127 | 41 | install -o root -g root -m 0644 files/upstart/defaults /etc/default/hadoop | 38 | install -o root -g root -m 0644 files/upstart/defaults /etc/default/hadoop |
129 | 42 | . /etc/default/hadoop $JAVA_HOME_PATH $HADOOP_INSTALLED $HOME_DIR $HADOOP_CONF_DIR | 39 | . /etc/default/hadoop $HADOOP_INSTALLED $HADOOP_CONF_DIR |
130 | 43 | fi | 40 | fi |
131 | 44 | juju-log "Configuring hadoop system environment - execute hadoopenv.sh " | 41 | juju-log "Configuring hadoop system environment - execute hadoopenv.sh " |
132 | 45 | . /etc/profile.d/hadoopenv.sh | 42 | . /etc/profile.d/hadoopenv.sh |
133 | 46 | } | 43 | } |
134 | 44 | |||
135 | 45 | configure_systemenv | ||
136 | 46 | |||
137 | 47 | |||
138 | 47 | # Helpers to support conditional restarts based | 48 | # Helpers to support conditional restarts based |
139 | 48 | # on specific files changing | 49 | # on specific files changing |
140 | 49 | 50 | ||
141 | @@ -68,7 +69,7 @@ | |||
142 | 68 | configure_sources () { | 69 | configure_sources () { |
143 | 69 | source=`config-get source` | 70 | source=`config-get source` |
144 | 70 | juju-log "Configuring hadoop using the Hadoop Ubuntu Team PPA..." | 71 | juju-log "Configuring hadoop using the Hadoop Ubuntu Team PPA..." |
146 | 71 | # add-apt-repository ppa:hadoop-ubuntu/$source | 72 | # add-apt-repository ppa:hadoop-ubuntu/$source |
147 | 72 | # apt-get update -qqy | 73 | # apt-get update -qqy |
148 | 73 | } | 74 | } |
149 | 74 | 75 | ||
150 | @@ -94,7 +95,9 @@ | |||
151 | 94 | mkdir -p $HADOOP_DIR | 95 | mkdir -p $HADOOP_DIR |
152 | 95 | mkdir -p $HADOOP_TMP_DIR | 96 | mkdir -p $HADOOP_TMP_DIR |
153 | 96 | apt-get install -qqy openjdk-7-jdk | 97 | apt-get install -qqy openjdk-7-jdk |
154 | 98 | |||
155 | 97 | tar -xzf files/archives/$HADOOP_VERSION.tar.gz -C $HADOOP_DIR | 99 | tar -xzf files/archives/$HADOOP_VERSION.tar.gz -C $HADOOP_DIR |
156 | 100 | chown -R ubuntu:hadoop $HADOOP_DIR | ||
157 | 98 | 101 | ||
158 | 99 | 102 | ||
159 | 100 | } | 103 | } |
160 | @@ -194,8 +197,10 @@ | |||
161 | 194 | if [ ! -d /etc/hadoop/conf.juju ] | 197 | if [ ! -d /etc/hadoop/conf.juju ] |
162 | 195 | then | 198 | then |
163 | 196 | mkdir -p /etc/hadoop/conf.juju | 199 | mkdir -p /etc/hadoop/conf.juju |
164 | 200 | |||
165 | 197 | cp $HADOOP_INSTALLED/etc/hadoop/* /etc/hadoop/conf.juju | 201 | cp $HADOOP_INSTALLED/etc/hadoop/* /etc/hadoop/conf.juju |
166 | 198 | #cp -r $HADOOP_INSTALLED/etc/hadoop /etc/hadoop/conf.juju | 202 | #cp -r $HADOOP_INSTALLED/etc/hadoop /etc/hadoop/conf.juju |
167 | 203 | rm /etc/hadoop/conf.juju/slaves | ||
168 | 199 | update-alternatives --install /etc/hadoop/conf hadoop-conf \ | 204 | update-alternatives --install /etc/hadoop/conf hadoop-conf \ |
169 | 200 | /etc/hadoop/conf.juju 50 | 205 | /etc/hadoop/conf.juju 50 |
170 | 201 | cp /etc/hadoop/conf.juju/mapred-site.xml.template \ | 206 | cp /etc/hadoop/conf.juju/mapred-site.xml.template \ |
171 | @@ -209,9 +214,10 @@ | |||
172 | 209 | dotdee --setup /etc/hadoop/conf.juju/mapred-site.xml | 214 | dotdee --setup /etc/hadoop/conf.juju/mapred-site.xml |
173 | 210 | dotdee --setup /etc/hadoop/conf.juju/hadoop-env.sh | 215 | dotdee --setup /etc/hadoop/conf.juju/hadoop-env.sh |
174 | 211 | dotdee --setup /etc/hadoop/conf.juju/yarn-site.xml | 216 | dotdee --setup /etc/hadoop/conf.juju/yarn-site.xml |
176 | 212 | 217 | chown -R ubuntu:hadoop /etc/hadoop/conf.juju | |
177 | 213 | fi | 218 | fi |
178 | 214 | # add JAVA_HOME to hadoop-env.sh | 219 | # add JAVA_HOME to hadoop-env.sh |
179 | 220 | JAVA_HOME_PATH=$(sudo find /usr/ -name java-7-openjdk-*) | ||
180 | 215 | sed -ir 's|export JAVA_HOME=.*|export JAVA_HOME='$JAVA_HOME_PATH'|' $HADOOP_CONF_DIR/hadoop-env.sh | 221 | sed -ir 's|export JAVA_HOME=.*|export JAVA_HOME='$JAVA_HOME_PATH'|' $HADOOP_CONF_DIR/hadoop-env.sh |
181 | 216 | # Configure Heap Size | 222 | # Configure Heap Size |
182 | 217 | heap=`config-get heap` | 223 | heap=`config-get heap` |
183 | @@ -228,9 +234,6 @@ | |||
184 | 228 | config_element "dfs.namenode.handler.count" \ | 234 | config_element "dfs.namenode.handler.count" \ |
185 | 229 | "`config-get dfs_namenode_handler_count`" > \ | 235 | "`config-get dfs_namenode_handler_count`" > \ |
186 | 230 | $dir/11-dfs.namenode.handler.count | 236 | $dir/11-dfs.namenode.handler.count |
187 | 231 | # config_element "dfs.blocksize" \ | ||
188 | 232 | # "`config-get dfs_block_size`" > \ | ||
189 | 233 | # $dir/12-dfs.blocksize | ||
190 | 234 | config_element "dfs.datanode.max.transfer.threads" \ | 237 | config_element "dfs.datanode.max.transfer.threads" \ |
191 | 235 | "`config-get dfs_datanode_max_xcievers`" > \ | 238 | "`config-get dfs_datanode_max_xcievers`" > \ |
192 | 236 | $dir/13-dfs.datanode.max.transfer.threads | 239 | $dir/13-dfs.datanode.max.transfer.threads |
193 | @@ -246,6 +249,15 @@ | |||
194 | 246 | config_element "dfs.replication" \ | 249 | config_element "dfs.replication" \ |
195 | 247 | "`config-get dfs_replication`" > \ | 250 | "`config-get dfs_replication`" > \ |
196 | 248 | $dir/17-dfs.replication | 251 | $dir/17-dfs.replication |
197 | 252 | config_element "dfs.heartbeat.interval"\ | ||
198 | 253 | "`config-get dfs_heartbeat_interval`" > \ | ||
199 | 254 | $dir/18-dfs.heartbeat.interval | ||
200 | 255 | config_element "dfs.datanode.data.dir" \ | ||
201 | 256 | "`config-get hadoop_dir_base`/cache/hadoop/dfs/name" > \ | ||
202 | 257 | $dir/19-dfs.datanode.data.dir | ||
203 | 258 | config_element "dfs.namenode.heartbeat.recheck-interval"\ | ||
204 | 259 | "`config-get dfs_namenode_heartbeat_recheck_interval`" > \ | ||
205 | 260 | $dir/20-dfs.namenode.heartbeat.recheck-interval | ||
206 | 249 | dotdee --update /etc/hadoop/conf.juju/hdfs-site.xml || true | 261 | dotdee --update /etc/hadoop/conf.juju/hdfs-site.xml || true |
207 | 250 | 262 | ||
208 | 251 | # Configure Hadoop YARN | 263 | # Configure Hadoop YARN |
209 | @@ -290,7 +302,7 @@ | |||
210 | 290 | mkdir -p $dir/cache/hadoop | 302 | mkdir -p $dir/cache/hadoop |
211 | 291 | # We don't want to do this recursively since we may be reinstalling, in which case | 303 | # We don't want to do this recursively since we may be reinstalling, in which case |
212 | 292 | # users have their own cache/<username> directories which shouldn't be stolen | 304 | # users have their own cache/<username> directories which shouldn't be stolen |
214 | 293 | groupadd -f hadoop | 305 | #groupadd -f hadoop |
215 | 294 | chown ubuntu:hadoop $dir $dir/cache $dir/cache/hadoop | 306 | chown ubuntu:hadoop $dir $dir/cache $dir/cache/hadoop |
216 | 295 | # Ensure group write on this directory or we can start namenode/datanode | 307 | # Ensure group write on this directory or we can start namenode/datanode |
217 | 296 | chmod 775 $dir/cache/hadoop | 308 | chmod 775 $dir/cache/hadoop |
218 | @@ -302,6 +314,7 @@ | |||
219 | 302 | datanode|secondarynamenode) | 314 | datanode|secondarynamenode) |
220 | 303 | sshkey=`relation-get namenode_sshkey` | 315 | sshkey=`relation-get namenode_sshkey` |
221 | 304 | echo $sshkey >> /home/ubuntu/.ssh/authorized_keys | 316 | echo $sshkey >> /home/ubuntu/.ssh/authorized_keys |
222 | 317 | chown -R ubuntu:hadoop /home/ubuntu/.ssh | ||
223 | 305 | ;; | 318 | ;; |
224 | 306 | namenode) | 319 | namenode) |
225 | 307 | relation-set namenode_sshkey="$(cat /home/ubuntu/.ssh/id_rsa.pub)" | 320 | relation-set namenode_sshkey="$(cat /home/ubuntu/.ssh/id_rsa.pub)" |
226 | @@ -311,15 +324,13 @@ | |||
227 | 311 | juju-log "Configuring service unit relation $1..." | 324 | juju-log "Configuring service unit relation $1..." |
228 | 312 | case $1 in | 325 | case $1 in |
229 | 313 | datanode|secondarynamenode|mapred-namenode) | 326 | datanode|secondarynamenode|mapred-namenode) |
232 | 314 | # sshkey=`relation-get namenode_sshkey` | 327 | |
231 | 315 | # echo $sshkey >> /home/ubuntu/.ssh/authorized_keys | ||
233 | 316 | namenode_address=`relation-get private-address` | 328 | namenode_address=`relation-get private-address` |
234 | 317 | config_element "fs.default.name" "hdfs://$namenode_address:9000" > \ | 329 | config_element "fs.default.name" "hdfs://$namenode_address:9000" > \ |
235 | 318 | $dir/50-fs.default.name | 330 | $dir/50-fs.default.name |
236 | 319 | ;; | 331 | ;; |
237 | 320 | namenode) | 332 | namenode) |
238 | 321 | private_address=`unit-get private-address` | 333 | private_address=`unit-get private-address` |
239 | 322 | # relation-set namenode_sshkey="$(cat /home/ubuntu/.ssh/id_rsa.pub)" | ||
240 | 323 | config_element "fs.default.name" "hdfs://$private_address:9000" > \ | 334 | config_element "fs.default.name" "hdfs://$private_address:9000" > \ |
241 | 324 | $dir/50-fs.default.name | 335 | $dir/50-fs.default.name |
242 | 325 | ;; | 336 | ;; |
243 | @@ -384,7 +395,7 @@ | |||
244 | 384 | 395 | ||
245 | 385 | format_namenode () { | 396 | format_namenode () { |
246 | 386 | juju-log "Formatting namenode filesystem" | 397 | juju-log "Formatting namenode filesystem" |
248 | 387 | hdfs namenode -format | 398 | su ubuntu -c "$HADOOP_INSTALLED/bin/hdfs namenode -format" |
249 | 388 | chown -R ubuntu:hadoop $HADOOP_DIR | 399 | chown -R ubuntu:hadoop $HADOOP_DIR |
250 | 389 | } | 400 | } |
251 | 390 | 401 | ||
252 | @@ -397,16 +408,18 @@ | |||
253 | 397 | juju-log "Restarting hdfs: $hdfs_role" | 408 | juju-log "Restarting hdfs: $hdfs_role" |
254 | 398 | case $hdfs_role in | 409 | case $hdfs_role in |
255 | 399 | namenode) | 410 | namenode) |
258 | 400 | hadoop-daemon.sh stop namenode | 411 | #hadoop-daemon.sh stop namenode |
259 | 401 | hadoop-daemon.sh start namenode | 412 | #hadoop-daemon.sh start namenode |
260 | 413 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop namenode" | ||
261 | 414 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start namenode" | ||
262 | 402 | ;; | 415 | ;; |
263 | 403 | datanode) | 416 | datanode) |
266 | 404 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemons.sh stop datanode" | 417 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs stop datanode" |
267 | 405 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemons.sh start datanode" | 418 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script hdfs start datanode" |
268 | 406 | ;; | 419 | ;; |
269 | 407 | secondarynamenode) | 420 | secondarynamenode) |
272 | 408 | hadoop-daemons.sh stop secondarynamenode | 421 | # will be replace with HA hadoop-daemons.sh stop secondarynamenode |
273 | 409 | hadoop-daemons.sh start secondarynamenode | 422 | # hadoop-daemons.sh start secondarynamenode |
274 | 410 | ;; | 423 | ;; |
275 | 411 | esac | 424 | esac |
276 | 412 | } | 425 | } |
277 | @@ -414,12 +427,14 @@ | |||
278 | 414 | juju-log "Restarting yarn: $mapred_role" | 427 | juju-log "Restarting yarn: $mapred_role" |
279 | 415 | case $mapred_role in | 428 | case $mapred_role in |
280 | 416 | resourcemanager) | 429 | resourcemanager) |
283 | 417 | yarn-daemon.sh stop resourcemanager | 430 | #yarn-daemon.sh stop resourcemanager |
284 | 418 | yarn-daemon.sh start resourcemanager | 431 | #yarn-daemon.sh start resourcemanager |
285 | 432 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager" | ||
286 | 433 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start resourcemanager" | ||
287 | 419 | ;; | 434 | ;; |
288 | 420 | nodemanager) | 435 | nodemanager) |
291 | 421 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemons.sh stop nodemanager" | 436 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager" |
292 | 422 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemons.sh start nodemanager" | 437 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR start nodemanager" |
293 | 423 | ;; | 438 | ;; |
294 | 424 | esac | 439 | esac |
295 | 425 | } | 440 | } |
296 | @@ -427,10 +442,7 @@ | |||
297 | 427 | 442 | ||
298 | 428 | # Hadoop Service Control Commands | 443 | # Hadoop Service Control Commands |
299 | 429 | restart_hadoop () { | 444 | restart_hadoop () { |
304 | 430 | # [ "$hdfs_role" != "unconfigured" ] && \ | 445 | |
301 | 431 | # _restart_ hadoop-$hdfs_role || : | ||
302 | 432 | # [ "$mapred_role" != "unconfigured" ] && \ | ||
303 | 433 | # _restart_ hadoop-$mapred_role || : | ||
305 | 434 | juju-log "restart_hadoop hdfs_role=$hdfs_role and yarn_role=$mapred_role" | 446 | juju-log "restart_hadoop hdfs_role=$hdfs_role and yarn_role=$mapred_role" |
306 | 435 | [ "$hdfs_role" != "unconfigured" ] && \ | 447 | [ "$hdfs_role" != "unconfigured" ] && \ |
307 | 436 | _restart_hdfs || : | 448 | _restart_hdfs || : |
308 | @@ -438,35 +450,32 @@ | |||
309 | 438 | [ "$mapred_role" != "unconfigured" ] && \ | 450 | [ "$mapred_role" != "unconfigured" ] && \ |
310 | 439 | _restart_yarn || : | 451 | _restart_yarn || : |
311 | 440 | } | 452 | } |
312 | 453 | |||
313 | 441 | stop_hadoop () { | 454 | stop_hadoop () { |
318 | 442 | # [ "$hdfs_role" != "unconfigured" ] && \ | 455 | |
315 | 443 | # _stop_ hadoop-$hdfs_role || : | ||
316 | 444 | # [ "$mapred_role" != "unconfigured" ] && \ | ||
317 | 445 | # _stop_ hadoop-$mapred_role || : | ||
319 | 446 | juju-log "Stop_hadoop hdfs_role=$hdfs_role and yarn_role=$mapred_role" | 456 | juju-log "Stop_hadoop hdfs_role=$hdfs_role and yarn_role=$mapred_role" |
320 | 447 | case $hdfs_role in | 457 | case $hdfs_role in |
321 | 448 | namenode) | 458 | namenode) |
323 | 449 | hadoop-daemon.sh stop namenode | 459 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop namenode" |
324 | 450 | ;; | 460 | ;; |
325 | 451 | datanode) | 461 | datanode) |
327 | 452 | hadoop-daemons.sh stop datanode | 462 | su ubuntu -c "$HADOOP_INSTALLED/sbin/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop datanode" |
328 | 453 | ;; | 463 | ;; |
329 | 454 | esac | 464 | esac |
330 | 465 | |||
331 | 455 | case $mapred_role in | 466 | case $mapred_role in |
332 | 456 | resourcemanager) | 467 | resourcemanager) |
334 | 457 | yarn-daemon.sh stop resourcemanager | 468 | #yarn-daemon.sh stop resourcemanager |
335 | 469 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop resourcemanager" | ||
336 | 458 | ;; | 470 | ;; |
337 | 459 | nodemanager) | 471 | nodemanager) |
339 | 460 | yarn-daemons.sh stop nodemanager | 472 | su ubuntu -c "$HADOOP_INSTALLED/sbin/yarn-daemon.sh --config $HADOOP_CONF_DIR stop nodemanager" |
340 | 461 | ;; | 473 | ;; |
341 | 462 | esac | 474 | esac |
342 | 475 | |||
343 | 476 | |||
344 | 463 | } | 477 | } |
351 | 464 | #start_hadoop () { | 478 | |
346 | 465 | # [ "$hdfs_role" != "unconfigured" ] && \ | ||
347 | 466 | # _start_ hadoop-$hdfs_role || : | ||
348 | 467 | # [ "$mapred_role" != "unconfigured" ] && \ | ||
349 | 468 | # _start_ hadoop-$mapred_role || : | ||
350 | 469 | #} | ||
352 | 470 | 479 | ||
353 | 471 | # Restart services only if configuration files | 480 | # Restart services only if configuration files |
354 | 472 | # have actually changed | 481 | # have actually changed |
355 | @@ -484,22 +493,39 @@ | |||
356 | 484 | then | 493 | then |
357 | 485 | # Just restart HDFS role | 494 | # Just restart HDFS role |
358 | 486 | [ "$hdfs_role" != "unconfigured" ] && \ | 495 | [ "$hdfs_role" != "unconfigured" ] && \ |
359 | 487 | # _restart_ hadoop-$hdfs_role || : | ||
360 | 488 | _restart_hdfs || : | 496 | _restart_hdfs || : |
361 | 489 | fi | 497 | fi |
362 | 490 | if file_changed /etc/hadoop/conf.juju/yarn-site.xml | 498 | if file_changed /etc/hadoop/conf.juju/yarn-site.xml |
363 | 491 | then | 499 | then |
364 | 492 | # Just restart mapreduce role | 500 | # Just restart mapreduce role |
365 | 493 | [ "$mapred_role" != "unconfigured" ] && \ | 501 | [ "$mapred_role" != "unconfigured" ] && \ |
366 | 494 | # _restart_ hadoop-$mapred_role || : | ||
367 | 495 | _restart_yarn || : | 502 | _restart_yarn || : |
368 | 496 | fi | 503 | fi |
369 | 497 | } | 504 | } |
370 | 505 | install_elasticsearch_hadoop () { | ||
371 | 506 | if [ ! -f "/home/ubuntu/elasticsearch-hadoop-2.0.0/dist/elasticsearch-hadoop-2.0.0.jar" ] ; | ||
372 | 507 | then | ||
373 | 508 | apt-get install unzip -y | ||
374 | 509 | cd /home/ubuntu | ||
375 | 510 | |||
376 | 511 | wget http://download.elasticsearch.org/hadoop/elasticsearch-hadoop-2.0.0{.zip,.zip.sha1.txt} | ||
377 | 512 | sh1=$(openssl sha1 *.zip | awk '{print $NF}') | ||
378 | 513 | sh2=$(cat *.sha1.txt | awk '{print $1;}') | ||
379 | 514 | if [ $sh1 != $sh2 ] ; | ||
380 | 515 | then | ||
381 | 516 | juju-log "invalid checksum" | ||
382 | 517 | exit 1 | ||
383 | 518 | fi | ||
384 | 519 | |||
385 | 520 | unzip elasticsearch-hadoop-2.0.0.zip | ||
386 | 521 | elk_jar_path="export LIBJARS=\$LIBJARS:/home/ubuntu/elasticsearch-hadoop-2.0.0/dist/elasticsearch-hadoop-2.0.0.jar" | ||
387 | 522 | echo $elk_jar_path >> /home/ubuntu/.bashrc | ||
388 | 523 | |||
389 | 524 | fi | ||
390 | 525 | } | ||
391 | 498 | 526 | ||
392 | 499 | install_job () { | 527 | install_job () { |
396 | 500 | #HADOOP_HOME=/usr/lib/hadoop | 528 | |
394 | 501 | set_hadoopenv | ||
395 | 502 | configure_systemenv | ||
397 | 503 | juju-log "installing terasort script" | 529 | juju-log "installing terasort script" |
398 | 504 | cp scripts/terasort.sh $HADOOP_DIR | 530 | cp scripts/terasort.sh $HADOOP_DIR |
399 | 505 | chown ubuntu.hadoop $HADOOP_DIR/terasort.sh | 531 | chown ubuntu.hadoop $HADOOP_DIR/terasort.sh |
400 | @@ -535,31 +561,27 @@ | |||
401 | 535 | install) | 561 | install) |
402 | 536 | configure_hosts | 562 | configure_hosts |
403 | 537 | configure_sources | 563 | configure_sources |
405 | 538 | set_hadoopenv | 564 | |
406 | 539 | install_base_packages | 565 | install_base_packages |
408 | 540 | configure_systemenv | 566 | |
409 | 541 | install_optional_packages | 567 | install_optional_packages |
410 | 542 | configure_hadoop | 568 | configure_hadoop |
411 | 543 | configure_tmp_dir_perms | 569 | configure_tmp_dir_perms |
412 | 544 | ;; | 570 | ;; |
413 | 571 | stop) | ||
414 | 572 | stop_hadoop | ||
415 | 573 | ;; | ||
416 | 545 | resourcemanager-relation-joined) | 574 | resourcemanager-relation-joined) |
417 | 546 | case $mapred_role in | 575 | case $mapred_role in |
418 | 547 | unconfigured) | 576 | unconfigured) |
419 | 548 | juju-log "Configuring this unit as a resourcemanager" | 577 | juju-log "Configuring this unit as a resourcemanager" |
424 | 549 | mapred_role="resourcemanager" | 578 | mapred_role="resourcemanager" |
421 | 550 | # this is a hack, should be removed with .dep package install | ||
422 | 551 | #mkdir -p /usr/share/doc/hadoop-resourcemanager | ||
423 | 552 | # end of hack | ||
425 | 553 | configure_role_relation $mapred_role | 579 | configure_role_relation $mapred_role |
429 | 554 | install_packages $mapred_role | 580 | install_packages $mapred_role |
427 | 555 | set_hadoopenv | ||
428 | 556 | configure_systemenv | ||
430 | 557 | _restart_yarn | 581 | _restart_yarn |
431 | 558 | open_ports | 582 | open_ports |
432 | 559 | install_job | 583 | install_job |
436 | 560 | # Some hadoop processes take a bit of time to start | 584 | |
434 | 561 | # we need to let them get to a point where they are | ||
435 | 562 | # ready to accept connections | ||
437 | 563 | sleep 10 && relation-set ready="true" | 585 | sleep 10 && relation-set ready="true" |
438 | 564 | ;; | 586 | ;; |
439 | 565 | resourcemanager) | 587 | resourcemanager) |
440 | @@ -592,8 +614,7 @@ | |||
441 | 592 | # end of hack | 614 | # end of hack |
442 | 593 | configure_role_relation $mapred_role | 615 | configure_role_relation $mapred_role |
443 | 594 | install_packages $mapred_role | 616 | install_packages $mapred_role |
446 | 595 | set_hadoopenv | 617 | |
445 | 596 | configure_systemenv | ||
447 | 597 | _restart_yarn | 618 | _restart_yarn |
448 | 598 | open_ports | 619 | open_ports |
449 | 599 | fi | 620 | fi |
450 | @@ -632,8 +653,7 @@ | |||
451 | 632 | # this is a hack, should be removed with .dep package install | 653 | # this is a hack, should be removed with .dep package install |
452 | 633 | #mkdir -p /usr/share/doc/hadoop-namenode | 654 | #mkdir -p /usr/share/doc/hadoop-namenode |
453 | 634 | # end of hack | 655 | # end of hack |
456 | 635 | set_hadoopenv | 656 | |
455 | 636 | configure_systemenv | ||
457 | 637 | configure_role_relation $hdfs_role | 657 | configure_role_relation $hdfs_role |
458 | 638 | install_packages $hdfs_role | 658 | install_packages $hdfs_role |
459 | 639 | stop_hadoop | 659 | stop_hadoop |
460 | @@ -675,8 +695,7 @@ | |||
461 | 675 | # end of hack | 695 | # end of hack |
462 | 676 | configure_role_relation $hdfs_role | 696 | configure_role_relation $hdfs_role |
463 | 677 | install_packages $hdfs_role | 697 | install_packages $hdfs_role |
466 | 678 | set_hadoopenv | 698 | |
465 | 679 | configure_systemenv | ||
467 | 680 | _restart_hdfs | 699 | _restart_hdfs |
468 | 681 | open_ports | 700 | open_ports |
469 | 682 | fi | 701 | fi |
470 | @@ -703,14 +722,14 @@ | |||
471 | 703 | juju-log "Namenode not yet ready" | 722 | juju-log "Namenode not yet ready" |
472 | 704 | exit 0 | 723 | exit 0 |
473 | 705 | else | 724 | else |
474 | 725 | relation-set slave_IP=`unit-get private-address` | ||
475 | 706 | juju-log "Configuring this unit as a datanode" | 726 | juju-log "Configuring this unit as a datanode" |
476 | 707 | hdfs_role="datanode" | 727 | hdfs_role="datanode" |
477 | 708 | # this is a hack, should be removed with .dep package install | 728 | # this is a hack, should be removed with .dep package install |
478 | 709 | # mkdir -p /usr/share/doc/hadoop-datanode | 729 | # mkdir -p /usr/share/doc/hadoop-datanode |
479 | 710 | # end of hack | 730 | # end of hack |
480 | 711 | configure_role_relation $hdfs_role | 731 | configure_role_relation $hdfs_role |
483 | 712 | set_hadoopenv | 732 | |
482 | 713 | configure_systemenv | ||
484 | 714 | install_packages $hdfs_role | 733 | install_packages $hdfs_role |
485 | 715 | _restart_hdfs | 734 | _restart_hdfs |
486 | 716 | open_ports | 735 | open_ports |
487 | @@ -729,6 +748,11 @@ | |||
488 | 729 | ;; | 748 | ;; |
489 | 730 | esac | 749 | esac |
490 | 731 | ;; | 750 | ;; |
491 | 751 | elasticsearch-relation-joined) | ||
492 | 752 | juju-log "Reconfiguring this resourcemanager to communicate with elasticsearch cluster" | ||
493 | 753 | [ mapred_role="resourcemanager" ] && install_elasticsearch_hadoop || : | ||
494 | 754 | |||
495 | 755 | ;; | ||
496 | 732 | ganglia-relation-changed) | 756 | ganglia-relation-changed) |
497 | 733 | # Call generic ganglia install and configure script | 757 | # Call generic ganglia install and configure script |
498 | 734 | # TODO supercede when subordinates land. | 758 | # TODO supercede when subordinates land. |
499 | @@ -741,14 +765,26 @@ | |||
500 | 741 | conditional_restart # only restart if pertinent config has changed! | 765 | conditional_restart # only restart if pertinent config has changed! |
501 | 742 | ;; | 766 | ;; |
502 | 743 | upgrade-charm|config-changed) | 767 | upgrade-charm|config-changed) |
504 | 744 | set_hadoopenv | 768 | |
505 | 745 | install_optional_packages | 769 | install_optional_packages |
506 | 746 | configure_systemenv | ||
507 | 747 | configure_hadoop | 770 | configure_hadoop |
508 | 748 | configure_tmp_dir_perms | 771 | configure_tmp_dir_perms |
509 | 749 | conditional_restart # only restart if pertinent config has changed! | 772 | conditional_restart # only restart if pertinent config has changed! |
510 | 750 | open_ports | 773 | open_ports |
511 | 751 | ;; | 774 | ;; |
512 | 775 | namenode-relation-changed) | ||
513 | 776 | juju-log "Configuring namenode - Changed phase" | ||
514 | 777 | slave_IP=`relation-get slave_IP` | ||
515 | 778 | if [ -z "$slave_IP" ] | ||
516 | 779 | then | ||
517 | 780 | juju-log "Configuring namenode changed - Changed phase - NO slave ip" | ||
518 | 781 | exit 0 | ||
519 | 782 | else | ||
520 | 783 | juju-log "Configuring namenode - Changed phase - got slave_ip= $slave_IP" | ||
521 | 784 | echo $slave_IP>>/etc/hadoop/conf.juju/slaves | ||
522 | 785 | chown -R ubuntu:hadoop /etc/hadoop/conf.juju | ||
523 | 786 | fi | ||
524 | 787 | ;; | ||
525 | 752 | *) | 788 | *) |
526 | 753 | juju-log "Command not recognised" | 789 | juju-log "Command not recognised" |
527 | 754 | ;; | 790 | ;; |
528 | 755 | 791 | ||
529 | === modified file 'metadata.yaml' | |||
530 | --- metadata.yaml 2014-06-13 22:40:33 +0000 | |||
531 | +++ metadata.yaml 2014-07-07 14:22:54 +0000 | |||
532 | @@ -21,3 +21,5 @@ | |||
533 | 21 | interface: mapred | 21 | interface: mapred |
534 | 22 | mapred-namenode: | 22 | mapred-namenode: |
535 | 23 | interface: dfs | 23 | interface: dfs |
536 | 24 | elasticsearch: | ||
537 | 25 | interface: elasticsearch |
Amir,
I haven't run a deploy test yet. I've noted the one thing that stood out to me. A possible note here about interface names - since this just relates to elasticsearch and is not hyper specific to the E.lasticsearch L.ogstash K.ibana stack; it may make more sense to name the relationship appropriately, as this relationship would work with any data river associated with Elastic Search.
Otherwise, looks good so far. I'll run a deployment test once you've pushed the modifications.