Merge lp:~admcleod/charms/trusty/apache-hadoop-hdfs-master/hadoop-upgrade into lp:charms/trusty/apache-hadoop-hdfs-master

[Andrew McLeod]

Added hadoop-pre-upgrade, hadoop-upgrade and hadoop-post-upgrade actions, modified resources.yaml

[lp:~admcleod/charms/trusty/apache-hadoop-hdfs-master/hadoop-upgrade]

96. By Andrew McLeod on 2015-10-22

removed some unnecessary comments

Unmerged revisions

96. By Andrew McLeod on 2015-10-22

removed some unnecessary comments

95. By Andrew McLeod on 2015-10-22

updated README.md

94. By Andrew McLeod on 2015-10-22

wait for hdfs to come out of safemode before deleting checkpoint file

93. By Andrew McLeod on 2015-10-21

trivial text modifications

92. By Andrew McLeod on 2015-10-20

hadoop upgrade actions additions etc

91. By Andrew McLeod on 2015-10-09

dist.yaml 2.7.1 - unstable (untested) hadoop

=== modified file 'README.md'
--- README.md	2015-10-07 20:35:26 +0000
+++ README.md	2015-10-22 16:54:30 +0000
@@ -74,6 +74,141 @@
 running jobs when enabling monitoring.
 
 
+## Upgrading
+
+This charm supports upgrading a non-HA hadoop cluster using the hdfs -upgrade
+flag via juju actions, from 2.4.1 to 2.7.1 and back.  Each unit that a hadoop
+service resides on must be upgraded - if the version of the hadoop software on a
+particular unit is different to one which it has a relationship with, the
+relation will fail.
+
+The process to upgrade hadoop is:
+
+    1. stop yarn
+    2. stop all datanodes (and secondary namenode if necessary)
+    3. upgrade the hadoop software on all related nodes
+    4. upgrade the namenode
+    5. restart the namenode, secondary namenode and datanodes.  
+
+Some checks have been built into the actions, but you should review the actions
+and validate each step if you plan to run this against a mission critical
+production system, especially the backup and rollback options.   
+
+**Warning:** These actions do not copy or merge new hadoop version configuration.
+these upgrade actions leave the existing configuration in place. It is up to
+the hadoop administrator to make any configuration changes necessary.   
+
+### Upgrade Actions
+
+#### hadoop-pre-upgrade
+
+    juju action do namenode/0 hadoop-pre-upgrade version=2.7.1
+
+ * Should only be run on the namenode
+ * Version param must be supplied - version=2.4.1 or version=2.7.1
+ * Performs HDFS checks twice and checks there are no differences / no jobs
+ still running
+ * Performs fsck on HDFS
+ * Attempts to back up edit log and fs image from /tmp/hadoop-hdfs
+ * Cycles namenode dfs daemons to update and create new checkpoint
+ * Stops namenode dfs daemons
+
+
+#### hadoop-upgrade
+
+    juju action do othernode/0 hadoop-upgrade version=2.7.1 rollback=false
+
+ * Run on any unit with hadoop software installed *AFTER* hadoop-pre-upgrade on
+  the namenode so that any running daemons are stopped
+ * Services with different versions of hadoop will not work together
+ * Version param must be supplied - version=2.4.1 or version=2.7.1
+ * Rollback param must be supplied - rollback=[true|false]. If rollback is true,
+  symlink is recreated, no new files are created.
+
+
+#### hadoop-post-upgrade
+
+    juju action do namenode/0 hadoop-post-upgrade version=2.7.1 rollback=false finalize=false
+
+ * Should only be run on a namenode
+ * Rollback param must be supplied - rollback=[true|false]. If rollback is true,
+   symlink is recreated, no new files are created.
+ * Starts HDFS daemon with -upgrade or -rollback switch to convert fs to new
+   version
+ * Performs fsck on HDFS
+ * Creates namespace log and compares to pre-upgrade logs
+ * Creates dfsadmon -report log and compares to pre-upgrade logs
+ * Takes option param finalize=[true|false] - if false, does nothing. If true,
+ finalizes HDFS upgrade (removes backup) - ***if HDFS is not finalized, deleted
+ files remain (hidden) on the fs and do not free up space - rollback is no
+ longer possible after finalize***
+
+
+#### Example upgrade steps
+
+***Do not copy & paste*** the following, unless you are using the apache core batch
+processing bundle - it is intended to be an example of how you upgrade a running
+hadoop cluster. All of these upgrade actions provide action responses, i.e.
+visible with 'juju action fetch' and also juju extended status updates.
+
+    juju action do yarn-master/0 stop-yarn
+
+    juju action fetch --wait 0 action-id
+
+    juju action do hdfs-master/0 hadoop-pre-upgrade version=2.7.1
+
+    watch 'juju status --format=tabular'
+
+    juju action do secondary-namenode/0 hadoop-upgrade version=2.7.1 rollback=false&
+    juju action do plugin/0 hadoop-upgrade version=2.7.1 rollback=false&
+    juju action do compute-slave/0 hadoop-upgrade version=2.7.1 rollback=false&
+    juju action do compute-slave/1 hadoop-upgrade version=2.7.1 rollback=false&
+    juju action do compute-slave/2 hadoop-upgrade version=2.7.1 rollback=false&
+    juju action do yarn-master/0 hadoop-upgrade version=2.7.1 rollback=false&
+    juju action do hdfs-master/0 hadoop-upgrade version=2.7.1 rollback=false&
+
+    watch 'juju status --format=tabular'
+
+    juju action do hdfs-master/0 hadoop-post-upgrade rollback=false version=2.7.1
+
+    watch 'juju status --format=tabular'
+
+    juju action do yarn-master/0 start-yarn
+
+ Test cluster functionality:
+
+    juju action do plugin/0 terasort
+
+*Once you have tested your upgrade, you must choose whether to finalize or
+rollback HDFS.*
+
+#### Finalize instructions
+
+NOTE: If you do not finalize HDFS after an upgrade, files which are deleted
+from HDFS will remain hidden on the file system, and therefore you will not
+reclaim used space. You must finalize once you have decided you are happy to
+proceed with the upgraded version, or rollback to the previous version.
+
+After completing the upgrade procedure:
+
+      juju action do namenode/0 hadoop-post-upgrade version=2.7.1 rollback=false
+      finalize=true
+
+#### Rollback instructions
+
+WARNING: Rollback removes any files created since upgrade - this means
+reverting to the previous system state, not just a previous version of software.
+
+* Follow the same procedure you used to upgrade, but make sure
+version='previously installed version' (this will not work for an older version
+which was not previously installed)
+* For hadoop-upgrade, rollback flag is optional, and specifies whether to use
+existing 'previous ver' directory, or unpack source agains
+* For hadoop-post-upgrade, rollback flag is mandatory as it forces HDFS to
+rollback to the previous version and image
+
+      juju action do namenode/0 hadoop-post-upgrade version=2.4.1 rollback=true
+
 ## Deploying in Network-Restricted Environments
 
 The Apache Hadoop charms can be deployed in environments with limited network

=== modified file 'actions.yaml'
--- actions.yaml	2015-08-17 18:12:14 +0000
+++ actions.yaml	2015-10-22 16:54:30 +0000
@@ -6,3 +6,34 @@
     description: All of the HDFS processes can be restarted with this Juju action.
 smoke-test:
     description: Verify that HDFS is working by creating and removing a small file.
+hadoop-pre-upgrade:
+  description: prepare hadoop cluster for upgrade
+  params:
+    version:
+      type: string
+      descrption: destination hadoop version X.X.X
+  required: [version] 
+hadoop-upgrade:
+  description: upgrade (or roll back) hadoop to specified version
+  params:
+    version:
+      type: string
+      description: destination hadoop version X.X.X
+    rollback:
+      type: boolean 
+      description: true or false - defaults to false
+  required: [version, rollback]
+hadoop-post-upgrade:
+  descrption: complete upgrade on hdfs-master node
+  params:
+    version:
+      type: string
+      description: destination hadoop version X.X.X
+    finalize:
+      type: boolean
+      default: false
+      description: finalizes hadoop upgrade - wipes backup image and allows real file deletion
+    rollback:
+      type: boolean
+      description: true or false
+  required: [version, rollback]

=== added file 'actions/hadoop-post-upgrade'
--- actions/hadoop-post-upgrade	1970-01-01 00:00:00 +0000
+++ actions/hadoop-post-upgrade	2015-10-22 16:54:30 +0000
@@ -0,0 +1,87 @@
+#!/bin/bash
+export SAVEPATH=$PATH
+. /etc/environment
+export PATH=$PATH:$SAVEPATH
+export JAVA_HOME
+
+current_hadoop_ver=`/usr/lib/hadoop/bin/hadoop version|head -n1|awk '{print $2}'`
+new_hadoop_ver=`action-get version`
+rollback=`action-get rollback`
+finalize=`action-get finalize`
+
+# Following condition may seem unintuitive - but at this point, current_hadoop_ver should match new_hadoop_ver
+# as current_hadoop_ver should be the new version of hadoop software
+if [ "$new_hadoop_ver" != "$current_hadoop_ver" ] ; then
+        action-set result="Version specified is not installed on this system, aborting"
+        action-fail "Version specified is not installed on this system, aborting"
+        exit 1
+fi
+
+if [ "${finalize}" == "True" ] ; then
+          cd ${HADOOP_HOME}
+          su hdfs -c "bin/hdfs dfsadmin -finalizeUpgrade"
+          action-set finalized="true"
+          status-set active "Ready - upgrade finalized, rollback unavailable" 
+          exit 0
+fi
+
+if [ "${rollback}" == "True" ] ; then 
+          cd ${HADOOP_HOME}
+          status-set maintenance "Performing namenode rollback..." 
+
+          # following workaround for rollback to 2.4.1 to 2.7.1:
+          # echo y to prompt, rather than using start-dfs.sh -rollback due to 
+          # https://issues.apache.org/jira/browse/HDFS-8226
+          su hdfs -c "echo y|bin/hdfs namenode -rollback"
+
+          status-set maintenance "Performing datanode rollback..."
+          su hdfs -c "sbin/hadoop-daemons.sh --config $HADOOP_CONF_DIR --script 'hdfs' start datanode -rollback"
+          su hdfs -c "sbin/start-dfs.sh"
+          procedure="rollback"
+fi
+  
+if [ "${rollback}" == "False" ] ; then
+          cd ${HADOOP_HOME}
+          status-set maintenance "starting dfs in upgrade mode"
+          su hdfs -c "sbin/start-dfs.sh -upgrade"
+          procedure="upgrade"
+fi              
+                          
+cd ${HADOOP_HOME}
+
+status-set maintenance "creating dfsadmin report"
+su hdfs -c "bin/hdfs dfsadmin -report | grep -v DFS|grep -v contact|grep -v Capacity > /tmp/dfs-v-new-report-1.log "
+diff /tmp/dfs-v-old-report-2.log /tmp/dfs-v-new-report-1.log 
+if [ ! $? -eq 0 ] ; then
+        action-set newhadoop.diff-report="failed diff - check /tmp/dfs-v-*-report*"
+fi
+
+status-set maintenance "creating fs log"
+su hdfs -c "bin/hdfs dfs -ls -R / > /tmp/dfs-v-new-lsr-0.log"
+diff /tmp/dfs-v-new-lsr-0.log /tmp/dfs-v-old-lsr-2.log
+if [ ! $? -eq 0 ] ; then
+        action-set newhadoop.diff-lsr="fail"
+fi
+
+status-set maintenance "checking fs"
+su hdfs -c "bin/hdfs fsck / -files -blocks -locations > /tmp/dfs-v-new-fsck-1.log"
+diff /tmp/dfs-v-old-fsck-1.log /tmp/dfs-v-new-fsck-1.log
+if [ ! $? -eq 0 ] ; then
+        action-set newhadoop.diff-fsck="fail"
+fi
+
+hadoop_version=`/usr/lib/hadoop/bin/hadoop version|head -n1|awk '{print $2}'`
+
+status-set maintenance "looking in hdfs for ${procedure} checkpoint files..."
+if ! su hdfs -c "bin/hdfs dfs -test -e hdfs:///user/ubuntu/checkpoint_hadoop" ; then
+         action-set result="${procedure} checkpoint file hdfs:///user/ubuntu/checkpoint_hadoop not found - check fs manually"
+         status-set active "Unknown - see action results and check fs manually"
+         action-set hadoop.version="${hadoop_version}"
+else                
+         status-set maintenance "deleting checkpoint file"
+         sleep 30 
+         su hdfs -c "bin/hdfs dfs -rm hdfs:///user/ubuntu/checkpoint_hadoop"
+         action-set newhadoop.running="true"
+         action-set hadoop.version="${hadoop_version}"
+         status-set active "Ready - hadoop version ${hadoop_version} ${procedure} complete" 
+fi

=== added file 'actions/hadoop-pre-upgrade'
--- actions/hadoop-pre-upgrade	1970-01-01 00:00:00 +0000
+++ actions/hadoop-pre-upgrade	2015-10-22 16:54:30 +0000
@@ -0,0 +1,138 @@
+#!/bin/bash
+export JUJU_PATH=$PATH
+. /etc/environment
+export PATH=$PATH:$JUJU_PATH
+export JAVA_HOME
+
+rm -rf /tmp/dfs-v*
+cd "$HADOOP_HOME"
+
+current_hadoop_ver=`/usr/lib/hadoop/bin/hadoop version|head -n1|awk '{print $2}'`
+new_hadoop_ver=`action-get version`
+
+function clean_checkpoint {
+        su hdfs -c "hdfs dfs -rm hdfs:///user/ubuntu/checkpoint_hadoop"
+}
+
+status-set maintenance "creating upgrade checkpoint files in hdfs:///user/ubuntu/"
+su hdfs -c "date > /tmp/checkpoint_hadoop"
+if ! su hdfs -c "bin/hdfs dfs -put /tmp/checkpoint_hadoop hdfs:///user/ubuntu/" ; then
+        action-fail "checkpoint file(s) could not be created, hdfs not running? Aborting upgrade"
+        exit 1
+fi
+
+if [ "$new_hadoop_ver" == "$current_hadoop_ver" ] ; then
+        action-set result="Same version already installed, aborting"
+        action-fail "Same version already installed"
+        clean_checkpoint
+        exit 1 
+fi
+
+status-set maintenance "checking hdfs root..."
+su hdfs -c "bin/hdfs fsck / -files -blocks -locations > /tmp/dfs-v-old-fsck-1.log"
+exitcode=$?
+action-set exitcode.fsck-hdfs-1=$exitcode
+if [ ! $exitcode -eq 0 ] ; then 
+        action-fail "fsck exitcode not zero - check manually"
+        clean_checkpoint
+        exit 1
+fi
+
+status-set maintenance "logging hdfs namespace..."
+su hdfs -c "bin/hdfs dfs -ls -R / > /tmp/dfs-v-old-lsr-1.log"
+exitcode=$?
+action-set exitcode.dfsls-1=$exitcode
+if [ ! $exitcode -eq 0 ] ; then 
+        action-fail "could not ls -R dfs - check manually"
+        clean_checkpoint
+        exit 1
+fi
+
+status-set maintenance "creating dfsadmin report..."
+su hdfs -c "bin/hdfs dfsadmin -report | grep -v DFS|grep -v contact|grep -v Capacity > /tmp/dfs-v-old-report-1.log"
+exitcode=$?
+action-set exitcode.dfsadmin-report=$exitcode
+if [ ! $exitcode -eq 0 ] ; then 
+        action-fail "Error running dfsadmin -report - check manually"
+        clean_checkpoint
+        exit 1
+fi
+
+status-set maintenance "checking hdfs namespace log..."
+su hdfs -c "bin/hdfs dfs -ls -R / > /tmp/dfs-v-old-lsr-2.log"
+diff /tmp/dfs-v-old-lsr-1.log /tmp/dfs-v-old-lsr-2.log > /dev/null
+if [ ! $? -eq 0 ] ; then 
+        action-fail "dfs ls -R - differences found (/tmp/dfs-v-old-lsr-?.log), aborting pre-upgrade"
+        clean_checkpoint
+        exit 1
+        status-set active "dfs ls -R - differences found - aborting pre-upgrade"
+fi 
+action-set dfsls.diff="No differences found between 1st and 2nd dfs ls -R runs"
+
+status-set maintenance "checking dfsadmin report log..."
+su hdfs -c "bin/hdfs dfsadmin -report | grep -v DFS|grep -v contact|grep -v Capacity > /tmp/dfs-v-old-report-2.log"
+diff /tmp/dfs-v-old-report-1.log /tmp/dfs-v-old-report-2.log > /dev/null
+if [ ! $? -eq 0 ] ; then
+        status-set active "dfsadmin report - differences found (/tmp/dfs-v-old-report-?.log) aborting pre-upgrade"
+        clean_checkpoint
+        exit 1
+fi
+action-set dfsadminreport.diff="No differences found between 1st and 2nd dfsadmin -report runs"
+
+status-set maintenance "backing up edit and fsimage files to /home/hdfs/backup"
+if [ ! -d /home/hdfs/backup/ ] ; then
+        mkdir /home/hdfs/backup/
+fi
+
+if [ -d /tmp/hadoop-hdfs/dfs/namesecondary/current/ ] ; then
+        cp /tmp/hadoop-hdfs/dfs/namesecondary/current/edits* /home/hdfs/backup/
+        exitcode=$?
+        action-set exitcode.backup-edits=$exitcode
+        if [ ! $exitcode -eq 0 ] ; then
+                action-fail "Could not cp /tmp/hadoop-hdfs/dfs/namesecondary/current/edits* to /home/hdfs/backup/"
+                clean_checkpoint
+                exit 1
+        fi
+
+        cp /tmp/hadoop-hdfs/dfs/namesecondary/current/fsimage* /home/hdfs/backup/
+        exitcode=$?
+        action-set exitcode.backup-fsimage=$exitcode
+        if [ ! $exitcode -eq 0 ] ; then
+                action-fail "Could not cp /tmp/hadoop-hdfs/dfs/namesecondary/current/fsimage* to /home/hdfs/backup/"
+                clean_checkpoint 
+                exit 1
+        fi
+fi
+
+status-set maintenance "cycling dfs to create namespace checkpoint for rollback"
+su hdfs -c "sbin/stop-dfs.sh"
+exitcode=$?
+action-set exitcode.cycledfs-1=$exitcode
+if [ ! $exitcode -eq 0 ] ; then 
+        action-fail "Could not stop dfs - pre-upgrade aborted"
+        clean_checkpoint
+        exit 1
+fi
+
+su hdfs -c "sbin/start-dfs.sh"
+exitcode=$?
+action-set exitcode.cycledfs-2=$exitcode
+if [ ! $exitcode -eq 0 ] ; then
+        action-fail "Could not start dfs - pre-upgrade aborted"
+        clean_checkpoint
+        exit 1
+fi
+
+status-set maintenance "bringing DFS daemons down"
+su hdfs -c "sbin/stop-dfs.sh"
+exitcode=$?
+action-set exitcode.start-dfs=$exitcode
+if [ ! $exitcode -eq 0 ] ; then
+        action-fail "Could not stop dfs - pre-upgrade aborted"
+        clean_checkpoint
+        exit 1
+fi
+
+action-set pre-upgrade="ready for upgrade"
+action-set next-step="run hadoop-upgrade on all other hadoop nodes before hdfs-master - see hdfs-master README.md"
+status-set maintenance "hadoop-pre-upgrade complete - run hadoop-upgrade on all hadoop nodes"

=== added file 'actions/hadoop-upgrade'
--- actions/hadoop-upgrade	1970-01-01 00:00:00 +0000
+++ actions/hadoop-upgrade	2015-10-22 16:54:30 +0000
@@ -0,0 +1,78 @@
+#!/bin/bash
+export SAVEPATH=$PATH
+. /etc/environment
+export PATH=$PATH:$SAVEPATH
+export JAVA_HOME
+
+current_hadoop_ver=`/usr/lib/hadoop/bin/hadoop version|head -n1|awk '{print $2}'`
+new_hadoop_ver=`action-get version`
+cpu_arch=`lscpu|grep -i arch|awk '{print $2}'`
+rollback=`action-get rollback`
+
+
+if pgrep -f Dproc_namenode ; then 
+        action-set result="namenode process detected, upgrade aborted"
+        status-set active "namenode process detected, upgrade aborted" 
+        exit 1
+fi
+
+if [ "$new_hadoop_ver" == "$current_hadoop_ver" ] ; then 
+        action-set result="Same version already installed, aborting"
+        action-fail "Same version already installed"
+        exit 1
+fi
+
+if [ "${rollback}" == "True" ] ; then
+        if [ -d /usr/lib/hadoop-${new_hadoop_ver} ] ; then
+                rm /usr/lib/hadoop
+                ln -s /usr/lib/hadoop-${new_hadoop_ver} /usr/lib/hadoop
+                if [ -d /usr/lib/hadoop-${current_hadoop_ver}/logs ] ; then
+                        mv /usr/lib/hadoop-${current_hadoop_ver}/logs /usr/lib/hadoop/
+                fi
+                action-set newhadoop.rollback="successfully rolled back"
+                status-set active "Ready - rollback to ${new_hadoop_ver} complete - ready for hadoop-post-upgrade"
+                exit 0
+        else
+                action-set newhadoop.rollback="previous version not found, unpacking..."
+                status-set active "previous version not found, unpacking..."
+        fi
+fi
+                
+status-set maintenance "Fetching hadoop-${new_hadoop_ver}-${cpu_arch}"
+juju-resources fetch hadoop-${new_hadoop_ver}-${cpu_arch}
+if [ ! $? -eq 0 ] ; then
+        action-set newhadoop.fetch="fail"
+        exit 1
+fi
+action-set newhadoop.fetch="success"
+
+status-set maintenance "Verifying hadoop-${new_hadoop_ver}-${cpu_arch}"
+juju-resources verify hadoop-${new_hadoop_ver}-${cpu_arch}
+if [ ! $? -eq 0 ] ; then
+        action-set newhadoop.verify="fail"
+        exit 1
+fi
+action-set newhadoop.verify="success"
+
+new_hadoop_path=`juju-resources resource_path hadoop-${new_hadoop_ver}-${cpu_arch}`
+if [ -h /usr/lib/hadoop ] ; then
+       rm /usr/lib/hadoop
+fi
+
+mv /usr/lib/hadoop/ /usr/lib/hadoop-${current_hadoop_ver}
+ln -s /usr/lib/hadoop-${current_hadoop_ver}/ /usr/lib/hadoop
+current_hadoop_path=hadoop-${current_hadoop_ver}
+
+status-set maintenance "Extracting hadoop-${new_hadoop_ver}-${cpu_arch}"
+tar -zxvf ${new_hadoop_path} -C /usr/lib/
+if [ $? -eq 0 ] ; then
+        if [ -h /usr/lib/hadoop ] ; then
+                rm /usr/lib/hadoop
+        fi
+        ln -s /usr/lib/hadoop-${new_hadoop_ver} /usr/lib/hadoop
+fi
+if [ -d ${current_hadoop_path}/logs ] ; then
+        mv ${current_hadoop_path}/logs ${new_hadoop_path}/
+fi
+action-set result="complete"
+status-set maintenance "hadoop version ${new_hadoop_ver} installed - ready for hadoop-post-upgrade"

=== modified file 'resources.yaml'
--- resources.yaml	2015-10-07 20:35:26 +0000
+++ resources.yaml	2015-10-22 16:54:30 +0000
@@ -26,3 +26,11 @@
     url: https://s3.amazonaws.com/jujubigdata/apache/x86_64/hadoop-2.4.1-a790d39.tar.gz
     hash: a790d39baba3a597bd226042496764e0520c2336eedb28a1a3d5c48572d3b672
     hash_type: sha256
+  hadoop-2.4.1-x86_64:
+    url: https://s3.amazonaws.com/jujubigdata/apache/x86_64/hadoop-2.4.1-a790d39.tar.gz
+    hash: a790d39baba3a597bd226042496764e0520c2336eedb28a1a3d5c48572d3b672
+    hash_type: sha256
+  hadoop-2.7.1-x86_64:
+    url: http://mirrors.ukfast.co.uk/sites/ftp.apache.org/hadoop/common/hadoop-2.7.1/hadoop-2.7.1.tar.gz
+    hash: 991dc34ea42a80b236ca46ff5d207107bcc844174df0441777248fdb6d8c9aa0
+    hash_type: sha256