Merge lp:~aisrael/charms/trusty/apache-hadoop-client/benchmarks into lp:~bigdata-dev/charms/trusty/apache-hadoop-client/trunk

Proposed by Adam Israel on 2015-05-28
Status: Merged
Merge reported by: Cory Johns
Merged at revision: not available
Proposed branch: lp:~aisrael/charms/trusty/apache-hadoop-client/benchmarks
Merge into: lp:~bigdata-dev/charms/trusty/apache-hadoop-client/trunk
Diff against target: 266 lines (+213/-0)
8 files modified
README.md (+44/-0)
actions.yaml (+38/-0)
actions/parseTerasort.py (+54/-0)
actions/teragen (+21/-0)
actions/terasort (+49/-0)
hooks/benchmark-relation-changed (+3/-0)
hooks/install (+2/-0)
metadata.yaml (+2/-0)
To merge this branch: bzr merge lp:~aisrael/charms/trusty/apache-hadoop-client/benchmarks
Reviewer Review Type Date Requested Status
Juju Big Data Development 2015-05-28 Pending
Review via email: mp+260526@code.launchpad.net

Description of the change

This merge proposal adds support for benchmarking, and implements a 'terasort' benchmark. This adds two external dependencies: python-pip (which may already be installed via other requirements) and charm-benchmark, which is installed via pip.

To post a comment you must log in.
80. By Adam Israel on 2015-05-28

Add a Benchmarking section to the README

Cory Johns (johnsca) wrote :

Awesome! Thanks for this. See my two inline comments, though, regarding the /etc/environment issues you ran into.

Cory Johns (johnsca) wrote :

Merged this, with some modifications, into ~bigdata-dev.

Cory Johns (johnsca) wrote :

I should have clarified. We decided it made more sense to apply this to the apache-hadoop-plugin charm, rather than -client, as that now serves as the general connection point which was previously the role of -client.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2015-05-19 16:00:07 +0000
3+++ README.md 2015-05-28 21:05:22 +0000
4@@ -22,6 +22,50 @@
5 juju ssh client/0
6 hadoop jar my-job.jar
7
8+## Benchmarking
9+
10+ You can perform a terasort benchmark, in order to gauge performance of your environment:
11+
12+ $ juju action do apache-hadoop-client/0 terasort
13+ Action queued with id: cbd981e8-3400-4c8f-8df1-c39c55a7eae6
14+ $ juju action fetch --wait 0 cbd981e8-3400-4c8f-8df1-c39c55a7eae6
15+ results:
16+ meta:
17+ composite:
18+ direction: asc
19+ units: ms
20+ value: "206676"
21+ results:
22+ raw: '{"Total vcore-seconds taken by all map tasks": "439783", "Spilled Records":
23+ "30000000", "WRONG_LENGTH": "0", "Reduce output records": "10000000", "HDFS:
24+ Number of bytes read": "1000001024", "Total vcore-seconds taken by all reduce
25+ tasks": "50275", "Reduce input groups": "10000000", "Shuffled Maps ": "8", "FILE:
26+ Number of bytes written": "3128977482", "Input split bytes": "1024", "Total
27+ time spent by all reduce tasks (ms)": "50275", "FILE: Number of large read operations":
28+ "0", "Bytes Read": "1000000000", "Virtual memory (bytes) snapshot": "7688794112",
29+ "Launched map tasks": "8", "GC time elapsed (ms)": "11656", "Bytes Written":
30+ "1000000000", "FILE: Number of read operations": "0", "HDFS: Number of write
31+ operations": "2", "Total megabyte-seconds taken by all reduce tasks": "51481600",
32+ "Combine output records": "0", "HDFS: Number of bytes written": "1000000000",
33+ "Total time spent by all map tasks (ms)": "439783", "Map output records": "10000000",
34+ "Physical memory (bytes) snapshot": "2329722880", "FILE: Number of write operations":
35+ "0", "Launched reduce tasks": "1", "Reduce input records": "10000000", "Total
36+ megabyte-seconds taken by all map tasks": "450337792", "WRONG_REDUCE": "0",
37+ "HDFS: Number of read operations": "27", "Reduce shuffle bytes": "1040000048",
38+ "Map input records": "10000000", "Map output materialized bytes": "1040000048",
39+ "CPU time spent (ms)": "195020", "Merged Map outputs": "8", "FILE: Number of
40+ bytes read": "2080000144", "Failed Shuffles": "0", "Total time spent by all
41+ maps in occupied slots (ms)": "439783", "WRONG_MAP": "0", "BAD_ID": "0", "Rack-local
42+ map tasks": "2", "IO_ERROR": "0", "Combine input records": "0", "Map output
43+ bytes": "1020000000", "CONNECTION": "0", "HDFS: Number of large read operations":
44+ "0", "Total committed heap usage (bytes)": "1755840512", "Data-local map tasks":
45+ "6", "Total time spent by all reduces in occupied slots (ms)": "50275"}'
46+ status: completed
47+ timing:
48+ completed: 2015-05-28 20:55:50 +0000 UTC
49+ enqueued: 2015-05-28 20:53:41 +0000 UTC
50+ started: 2015-05-28 20:53:44 +0000 UTC
51+
52
53 ## Contact Information
54
55
56=== added directory 'actions'
57=== added file 'actions.yaml'
58--- actions.yaml 1970-01-01 00:00:00 +0000
59+++ actions.yaml 2015-05-28 21:05:22 +0000
60@@ -0,0 +1,38 @@
61+teragen:
62+ description: foo
63+ params:
64+ size:
65+ description: The number of 100 byte rows, default to 100MB of data to generate and sort
66+ type: string
67+ default: "10000000"
68+ indir:
69+ description: foo
70+ type: string
71+ default: 'tera_demo_in'
72+terasort:
73+ description: foo
74+ params:
75+ indir:
76+ description: foo
77+ type: string
78+ default: 'tera_demo_in'
79+ outdir:
80+ description: foo
81+ type: string
82+ default: 'tera_demo_out'
83+ size:
84+ description: The number of 100 byte rows, default to 100MB of data to generate and sort
85+ type: string
86+ default: "10000000"
87+ maps:
88+ description: The default number of map tasks per job. 1-20
89+ type: integer
90+ default: 1
91+ reduces:
92+ description: The default number of reduce tasks per job. Typically set to 99% of the cluster's reduce capacity, so that if a node fails the reduces can still be executed in a single wave. Try 1-20
93+ type: integer
94+ default: 1
95+ numtasks:
96+ description: How many tasks to run per jvm. If set to -1, there is no limit.
97+ type: integer
98+ default: 1
99
100=== added file 'actions/parseTerasort.py'
101--- actions/parseTerasort.py 1970-01-01 00:00:00 +0000
102+++ actions/parseTerasort.py 2015-05-28 21:05:22 +0000
103@@ -0,0 +1,54 @@
104+#!/usr/bin/env python
105+"""
106+Simple script to parse cassandra-stress' transaction results
107+and reformat them as JSON for sending back to juju
108+"""
109+import sys
110+import subprocess
111+import json
112+from charmhelpers.contrib.benchmark import Benchmark
113+import re
114+
115+
116+def action_set(key, val):
117+ action_cmd = ['action-set']
118+ if isinstance(val, dict):
119+ for k, v in val.iteritems():
120+ action_set('%s.%s' % (key, k), v)
121+ return
122+
123+ action_cmd.append('%s=%s' % (key, val))
124+ subprocess.check_call(action_cmd)
125+
126+
127+def parse_terasort_output():
128+ """
129+ Parse the output from terasort and set the action results:
130+
131+ """
132+
133+ results = {}
134+
135+ # Find all of the interesting things
136+ regex = re.compile('\t+(.*)=(.*)')
137+ for line in sys.stdin.readlines():
138+ m = regex.match(line)
139+ if m:
140+ results[m.group(1)] = m.group(2)
141+ action_set("results.raw", json.dumps(results))
142+
143+ # Calculate what's important
144+ if 'CPU time spent (ms)' in results:
145+ composite = int(results['CPU time spent (ms)']) + int(results['GC time elapsed (ms)'])
146+ Benchmark.set_composite_score(
147+ composite,
148+ 'ms',
149+ 'asc'
150+ )
151+ else:
152+ print "Invalid test results"
153+ print results
154+
155+
156+if __name__ == "__main__":
157+ parse_terasort_output()
158
159=== added file 'actions/teragen'
160--- actions/teragen 1970-01-01 00:00:00 +0000
161+++ actions/teragen 2015-05-28 21:05:22 +0000
162@@ -0,0 +1,21 @@
163+#!/bin/bash
164+set -eux
165+SIZE=`action-get size`
166+IN_DIR=`action-get indir`
167+
168+benchmark-start
169+
170+# I don't know why, but have to source /etc/environment before and after
171+# invoking the bash shell to get it working.
172+. /etc/environment
173+su ubuntu << EOF
174+. /etc/environment
175+if JAVA_HOME=${JAVA_HOME} hadoop fs -stat ${IN_DIR}; then
176+ JAVA_HOME=${JAVA_HOME} hadoop fs -rm -r -skipTrash ${IN_DIR} || true
177+fi
178+
179+JAVA_HOME=${JAVA_HOME} hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar teragen ${SIZE} ${IN_DIR}
180+
181+EOF
182+
183+benchmark-finish
184
185=== added file 'actions/terasort'
186--- actions/terasort 1970-01-01 00:00:00 +0000
187+++ actions/terasort 2015-05-28 21:05:22 +0000
188@@ -0,0 +1,49 @@
189+#!/bin/bash
190+IN_DIR=`action-get indir`
191+OUT_DIR=`action-get outdir`
192+SIZE=`action-get size`
193+OPTIONS=''
194+
195+MAPS=`action-get maps`
196+REDUCES=`action-get reduces`
197+NUMTASKS=`action-get numtasks`
198+
199+OPTIONS="${OPTIONS} -D mapreduce.job.maps=${MAPS}"
200+OPTIONS="${OPTIONS} -D mapreduce.job.reduces=${REDUCES}"
201+OPTIONS="${OPTIONS} -D mapreduce.job.jvm.numtasks=${NUMTASKS}"
202+
203+mkdir -p /opt/terasort
204+chown ubuntu:ubuntu /opt/terasort
205+run=`date +%s`
206+
207+# HACK: the environment reset below is munging the PATH
208+OLDPATH=$PATH
209+
210+
211+# I don't know why, but have to source /etc/environment before and after
212+# invoking the bash shell to get it working.
213+. /etc/environment
214+su ubuntu << EOF
215+. /etc/environment
216+
217+mkdir -p /opt/terasort/results/$run
218+
219+# If there's no data generated yet, create it using the action defaults
220+if ! JAVA_HOME=${JAVA_HOME} hadoop fs -stat ${IN_DIR} &> /dev/null; then
221+ JAVA_HOME=${JAVA_HOME} hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar teragen ${SIZE} ${IN_DIR} > /dev/null
222+
223+fi
224+
225+# If there's already sorted data, remove it
226+if JAVA_HOME=${JAVA_HOME} hadoop fs -stat ${OUT_DIR} &> /dev/null; then
227+ JAVA_HOME=${JAVA_HOME} hadoop fs -rm -r -skipTrash ${OUT_DIR} || true
228+fi
229+
230+benchmark-start
231+JAVA_HOME=${JAVA_HOME} hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples*.jar terasort ${OPTIONS} ${IN_DIR} ${OUT_DIR} &> /opt/terasort/results/$run/terasort.log
232+benchmark-finish
233+
234+EOF
235+PATH=$OLDPATH
236+
237+`cat /opt/terasort/results/$run/terasort.log | python $CHARM_DIR/actions/parseTerasort.py`
238
239=== added file 'hooks/benchmark-relation-changed'
240--- hooks/benchmark-relation-changed 1970-01-01 00:00:00 +0000
241+++ hooks/benchmark-relation-changed 2015-05-28 21:05:22 +0000
242@@ -0,0 +1,3 @@
243+#!/bin/bash
244+
245+relation-set benchmarks=terasort
246
247=== modified file 'hooks/install'
248--- hooks/install 2015-05-11 22:25:12 +0000
249+++ hooks/install 2015-05-28 21:05:22 +0000
250@@ -1,2 +1,4 @@
251 #!/bin/bash
252+apt-get install -y python-pip && pip install -U charm-benchmark
253+
254 hooks/status-set blocked "Please add relation to apache-hadoop-plugin"
255
256=== added symlink 'hooks/upgrade-charm'
257=== target is u'install'
258=== modified file 'metadata.yaml'
259--- metadata.yaml 2015-05-12 22:18:09 +0000
260+++ metadata.yaml 2015-05-28 21:05:22 +0000
261@@ -12,3 +12,5 @@
262 hadoop-plugin:
263 interface: hadoop-plugin
264 scope: container
265+ benchmark:
266+ interface: benchmark

Subscribers

People subscribed via source and target branches