Merge lp:~aisrael/charms/trusty/apache-hadoop-client/benchmarks into lp:~bigdata-dev/charms/trusty/apache-hadoop-client/trunk

[Adam Israel]

This merge proposal adds support for benchmarking, and implements a 'terasort' benchmark. This adds two external dependencies: python-pip (which may already be installed via other requirements) and charm-benchmark, which is installed via pip.

[lp:~aisrael/charms/trusty/apache-hadoop-client/benchmarks]

80. By Adam Israel on 2015-05-28

Add a Benchmarking section to the README

[Cory Johns (johnsca)]

Awesome! Thanks for this. See my two inline comments, though, regarding the /etc/environment issues you ran into.

[Cory Johns (johnsca)]

Merged this, with some modifications, into ~bigdata-dev.

[Cory Johns (johnsca)]

I should have clarified. We decided it made more sense to apply this to the apache-hadoop-plugin charm, rather than -client, as that now serves as the general connection point which was previously the role of -client.

=== modified file 'README.md'
--- README.md	2015-05-19 16:00:07 +0000
+++ README.md	2015-05-28 21:05:22 +0000
@@ -22,6 +22,50 @@
     juju ssh client/0
     hadoop jar my-job.jar
 
+## Benchmarking
+
+    You can perform a terasort benchmark, in order to gauge performance of your environment:
+
+        $ juju action do apache-hadoop-client/0 terasort
+        Action queued with id: cbd981e8-3400-4c8f-8df1-c39c55a7eae6
+        $ juju action fetch --wait 0 cbd981e8-3400-4c8f-8df1-c39c55a7eae6
+        results:
+          meta:
+            composite:
+              direction: asc
+              units: ms
+              value: "206676"
+          results:
+            raw: '{"Total vcore-seconds taken by all map tasks": "439783", "Spilled Records":
+              "30000000", "WRONG_LENGTH": "0", "Reduce output records": "10000000", "HDFS:
+              Number of bytes read": "1000001024", "Total vcore-seconds taken by all reduce
+              tasks": "50275", "Reduce input groups": "10000000", "Shuffled Maps ": "8", "FILE:
+              Number of bytes written": "3128977482", "Input split bytes": "1024", "Total
+              time spent by all reduce tasks (ms)": "50275", "FILE: Number of large read operations":
+              "0", "Bytes Read": "1000000000", "Virtual memory (bytes) snapshot": "7688794112",
+              "Launched map tasks": "8", "GC time elapsed (ms)": "11656", "Bytes Written":
+              "1000000000", "FILE: Number of read operations": "0", "HDFS: Number of write
+              operations": "2", "Total megabyte-seconds taken by all reduce tasks": "51481600",
+              "Combine output records": "0", "HDFS: Number of bytes written": "1000000000",
+              "Total time spent by all map tasks (ms)": "439783", "Map output records": "10000000",
+              "Physical memory (bytes) snapshot": "2329722880", "FILE: Number of write operations":
+              "0", "Launched reduce tasks": "1", "Reduce input records": "10000000", "Total
+              megabyte-seconds taken by all map tasks": "450337792", "WRONG_REDUCE": "0",
+              "HDFS: Number of read operations": "27", "Reduce shuffle bytes": "1040000048",
+              "Map input records": "10000000", "Map output materialized bytes": "1040000048",
+              "CPU time spent (ms)": "195020", "Merged Map outputs": "8", "FILE: Number of
+              bytes read": "2080000144", "Failed Shuffles": "0", "Total time spent by all
+              maps in occupied slots (ms)": "439783", "WRONG_MAP": "0", "BAD_ID": "0", "Rack-local
+              map tasks": "2", "IO_ERROR": "0", "Combine input records": "0", "Map output
+              bytes": "1020000000", "CONNECTION": "0", "HDFS: Number of large read operations":
+              "0", "Total committed heap usage (bytes)": "1755840512", "Data-local map tasks":
+              "6", "Total time spent by all reduces in occupied slots (ms)": "50275"}'
+        status: completed
+        timing:
+          completed: 2015-05-28 20:55:50 +0000 UTC
+          enqueued: 2015-05-28 20:53:41 +0000 UTC
+          started: 2015-05-28 20:53:44 +0000 UTC
+
 
 ## Contact Information
 
 
=== added directory 'actions'
=== added file 'actions.yaml'
--- actions.yaml	1970-01-01 00:00:00 +0000
+++ actions.yaml	2015-05-28 21:05:22 +0000
@@ -0,0 +1,38 @@
+teragen:
+    description: foo
+    params:
+        size:
+            description: The number of 100 byte rows, default to 100MB of data to generate and sort
+            type: string
+            default: "10000000"
+        indir:
+            description: foo
+            type: string
+            default: 'tera_demo_in'
+terasort:
+    description: foo
+    params:
+        indir:
+            description: foo
+            type: string
+            default: 'tera_demo_in'
+        outdir:
+            description: foo
+            type: string
+            default: 'tera_demo_out'
+        size:
+            description: The number of 100 byte rows, default to 100MB of data to generate and sort
+            type: string
+            default: "10000000"
+        maps:
+            description: The default number of map tasks per job. 1-20
+            type: integer
+            default: 1
+        reduces:
+            description: The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Try 1-20
+            type: integer
+            default: 1
+        numtasks:
+            description: How many tasks to run per jvm. If set to -1, there is no limit.
+            type: integer
+            default: 1
 
=== added file 'actions/parseTerasort.py'
--- actions/parseTerasort.py	1970-01-01 00:00:00 +0000
+++ actions/parseTerasort.py	2015-05-28 21:05:22 +0000
@@ -0,0 +1,54 @@
+#!/usr/bin/env python
+"""
+Simple script to parse cassandra-stress' transaction results
+and reformat them as JSON for sending back to juju
+"""
+import sys
+import subprocess
+import json
+from charmhelpers.contrib.benchmark import Benchmark
+import re
+
+
+def action_set(key, val):
+    action_cmd = ['action-set']
+    if isinstance(val, dict):
+        for k, v in val.iteritems():
+            action_set('%s.%s' % (key, k), v)
+        return
+
+    action_cmd.append('%s=%s' % (key, val))
+    subprocess.check_call(action_cmd)
+
+
+def parse_terasort_output():
+    """
+    Parse the output from terasort and set the action results:
+
+    """
+
+    results = {}
+
+    # Find all of the interesting things
+    regex = re.compile('\t+(.*)=(.*)')
+    for line in sys.stdin.readlines():
+        m = regex.match(line)
+        if m:
+            results[m.group(1)] = m.group(2)
+    action_set("results.raw", json.dumps(results))
+
+    # Calculate what's important
+    if 'CPU time spent (ms)' in results:
+        composite = int(results['CPU time spent (ms)']) + int(results['GC time elapsed (ms)'])
+        Benchmark.set_composite_score(
+            composite,
+            'ms',
+            'asc'
+        )
+    else:
+        print "Invalid test results"
+        print results
+
+
+if __name__ == "__main__":
+    parse_terasort_output()
 
=== added file 'actions/teragen'
--- actions/teragen	1970-01-01 00:00:00 +0000
+++ actions/teragen	2015-05-28 21:05:22 +0000
@@ -0,0 +1,21 @@
+#!/bin/bash
+set -eux
+SIZE=`action-get size`
+IN_DIR=`action-get indir`
+
+benchmark-start
+
+# I don't know why, but have to source /etc/environment before and after
+# invoking the bash shell to get it working.
+. /etc/environment
+su ubuntu << EOF
+. /etc/environment
+if JAVA_HOME=${JAVA_HOME} hadoop fs -stat ${IN_DIR}; then
+    JAVA_HOME=${JAVA_HOME} hadoop fs -rm -r -skipTrash ${IN_DIR} || true
+fi
+
+JAVA_HOME=${JAVA_HOME} hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar teragen ${SIZE} ${IN_DIR}
+
+EOF
+
+benchmark-finish
 
=== added file 'actions/terasort'
--- actions/terasort	1970-01-01 00:00:00 +0000
+++ actions/terasort	2015-05-28 21:05:22 +0000
@@ -0,0 +1,49 @@
+#!/bin/bash
+IN_DIR=`action-get indir`
+OUT_DIR=`action-get outdir`
+SIZE=`action-get size`
+OPTIONS=''
+
+MAPS=`action-get maps`
+REDUCES=`action-get reduces`
+NUMTASKS=`action-get numtasks`
+
+OPTIONS="${OPTIONS} -D mapreduce.job.maps=${MAPS}"
+OPTIONS="${OPTIONS} -D mapreduce.job.reduces=${REDUCES}"
+OPTIONS="${OPTIONS} -D mapreduce.job.jvm.numtasks=${NUMTASKS}"
+
+mkdir -p /opt/terasort
+chown ubuntu:ubuntu /opt/terasort
+run=`date +%s`
+
+# HACK: the environment reset below is munging the PATH
+OLDPATH=$PATH
+
+
+# I don't know why, but have to source /etc/environment before and after
+# invoking the bash shell to get it working.
+. /etc/environment
+su ubuntu << EOF
+. /etc/environment
+
+mkdir -p /opt/terasort/results/$run
+
+# If there's no data generated yet, create it using the action defaults
+if ! JAVA_HOME=${JAVA_HOME} hadoop fs -stat ${IN_DIR} &> /dev/null; then
+    JAVA_HOME=${JAVA_HOME} hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar teragen ${SIZE} ${IN_DIR} > /dev/null
+
+fi
+
+# If there's already sorted data, remove it
+if JAVA_HOME=${JAVA_HOME} hadoop fs -stat ${OUT_DIR} &> /dev/null; then
+    JAVA_HOME=${JAVA_HOME} hadoop fs -rm -r -skipTrash ${OUT_DIR} || true
+fi
+
+benchmark-start
+JAVA_HOME=${JAVA_HOME} hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar terasort ${OPTIONS} ${IN_DIR} ${OUT_DIR} &> /opt/terasort/results/$run/terasort.log
+benchmark-finish
+
+EOF
+PATH=$OLDPATH
+
+`cat /opt/terasort/results/$run/terasort.log | python $CHARM_DIR/actions/parseTerasort.py`
 
=== added file 'hooks/benchmark-relation-changed'
--- hooks/benchmark-relation-changed	1970-01-01 00:00:00 +0000
+++ hooks/benchmark-relation-changed	2015-05-28 21:05:22 +0000
@@ -0,0 +1,3 @@
+#!/bin/bash
+
+relation-set benchmarks=terasort
 
=== modified file 'hooks/install'
--- hooks/install	2015-05-11 22:25:12 +0000
+++ hooks/install	2015-05-28 21:05:22 +0000
@@ -1,2 +1,4 @@
 #!/bin/bash
+apt-get install -y python-pip && pip install -U charm-benchmark
+
 hooks/status-set blocked "Please add relation to apache-hadoop-plugin"
 
=== added symlink 'hooks/upgrade-charm'
=== target is u'install'
=== modified file 'metadata.yaml'
--- metadata.yaml	2015-05-12 22:18:09 +0000
+++ metadata.yaml	2015-05-28 21:05:22 +0000
@@ -12,3 +12,5 @@
   hadoop-plugin:
     interface: hadoop-plugin
     scope: container
+  benchmark:
+    interface: benchmark