Merge lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/readme into lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/trunk

[Cory Johns]

New READMEs and minor relation cleanups

[lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/readme]

42. By Cory Johns on 2015-03-11

Disambiguate the interface in README.dev

[amir sanjar (asanjar)]

we will also need to return hostname of computes nodes in future

review: Approve

=== added file 'README.dev.md'
--- README.dev.md	1970-01-01 00:00:00 +0000
+++ README.dev.md	2015-03-11 18:02:10 +0000
@@ -0,0 +1,72 @@
+## Overview
+
+This charm provides computation and storage resources for an Apache Hadoop
+deployment, and is intended to be used only as a part of that deployment.
+This document describes how this charm connects to and interacts with the
+other components of the deployment.
+
+
+## Provided Relations
+
+### datanode (interface: dfs-slave)
+
+This relation connects this charm to the apache-hadoop-hdfs-master charm.
+It is a bi-directional interface, with the following keys being exchanged:
+
+* Sent to hdfs-master:
+
+  * `private-address`: Address of this unit, to be registered as a DataNode
+
+* Received from hdfs-master:
+
+  * `private-address`: Address of the HDFS master unit, to provide the NameNode
+  * `ready`: A flag indicating that HDFS is ready to register DataNodes
+
+Ports will soon be added to both of these.
+
+
+### nodemanager (interface: mapred-slave)
+
+This relation connects this charm to the apache-hadoop-yarn-master charm.
+It is a bi-directional interface, with the following keys being exchanged:
+
+* Sent to yarn-master:
+
+  * `private-address`: Address of this unit, to be registered as a NodeManager
+
+* Received from yarn-master:
+
+  * `private-address`: Address of the YARN master unit, to provide the ResourceManager
+  * `ready`: A flag indicating that YARN is ready to register NodeManagers
+
+Ports will soon be added to both of these.
+
+
+## Required Relations
+
+*There are no required relations for this charm.*
+
+
+## Manual Deployment
+
+The easiest way to deploy the core Apache Hadoop platform is to use one of
+the [apache-core-batch-processing-* bundles](https://jujucharms.com/q/bigdata-dev/apache?type=bundle).
+However, to manually deploy the base Apache Hadoop platform without using one of the
+bundles, you can use the following:
+
+    juju deploy apache-hadoop-hdfs-master hdfs-master
+    juju deploy apache-hadoop-hdfs-secondary secondary-namenode
+    juju deploy apache-hadoop-yarn-master yarn-master
+    juju deploy apache-hadoop-compute-slave compute-slave -n3
+    juju deploy apache-hadoop-client client
+    juju add-relation yarn-master hdfs-master
+    juju add-relation secondary-namenode hdfs-master
+    juju add-relation compute-slave yarn-master
+    juju add-relation compute-slave hdfs-master
+    juju add-relation client yarn-master
+    juju add-relation client hdfs-master
+
+This will create a scalable deployment with separate nodes for each master,
+and a three unit compute slave (NodeManager and DataNode) cluster.  The master
+charms also support co-locating using the `--to` option to `juju deploy` for
+more dense deployments.

=== modified file 'README.md'
--- README.md	2015-02-13 22:34:21 +0000
+++ README.md	2015-03-11 18:02:10 +0000
@@ -1,118 +1,39 @@
 ## Overview
 
-This charm is a component of the Apache Hadoop platform.  It is intended
-to be deployed with the other components using the bundle:
-`bundle:~bigdata-charmers/apache-hadoop`
-
-**What is Apache Hadoop?**
-
 The Apache Hadoop software library is a framework that allows for the
 distributed processing of large data sets across clusters of computers
 using a simple programming model.
 
-It is designed to scale up from single servers to thousands of machines,
-each offering local computation and storage. Rather than rely on hardware
-to deliver high-avaiability, the library itself is designed to detect
-and handle failures at the application layer, so delivering a
-highly-availabile service on top of a cluster of computers, each of
-which may be prone to failures.
-
-Apache Hadoop 2.4.1 consists of significant improvements over the previous stable
-release (hadoop-1.x).
-
-Here is a short overview of the improvments to both HDFS and MapReduce.
-
- - **HDFS Federation**
-   In order to scale the name service horizontally, federation uses multiple
-   independent Namenodes/Namespaces. The Namenodes are federated, that is, the
-   Namenodes are independent   and don't require coordination with each other.
-   The datanodes are used as common storage for blocks by all the Namenodes.
-   Each datanode registers with all the Namenodes in the cluster.   Datanodes
-   send periodic heartbeats and block reports and handles commands from the
-   Namenodes.
-
-   More details are available in the HDFS Federation document:
-   <http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/Federation.html>
-
- - **MapReduce NextGen aka YARN aka MRv2**
-   The new architecture introduced in hadoop-0.23, divides the two major functions of the
-   JobTracker: resource management and job life-cycle management into separate components.
-   The new ResourceManager manages the global assignment of compute resources to
-   applications and the per-application ApplicationMaster manages the application‚
-   scheduling and coordination.
-   An application is either a single job in the sense of classic MapReduce jobs or a DAG of
-   such jobs.
-
-   The ResourceManager and per-machine NodeManager daemon, which manages the user
-   processes on   that machine, form the computation fabric.
-
-   The per-application ApplicationMaster is, in effect, a framework specific
-   library and is tasked with negotiating resources from the ResourceManager and
-   working with the NodeManager(s) to execute and monitor the tasks.
-
-   More details are available in the YARN document:
-   <http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/YARN.html>
+This charm deploys a compute / slave node running the NodeManager
+and DataNode components of
+[Apache Hadoop 2.4.1](http://hadoop.apache.org/docs/r2.4.1/),
+which provides computation and storage resources to the platform.
 
 ## Usage
 
-This charm manages the compute slave nodes, which include the DataNode and
-NodeManager components.  It is intended to be used with `apache-hadoop-hdfs-master`
-and `apache-hadoop-yarn-master`.
-
-### Simple Usage: Single YARN / HDFS master deployment
-
-In this configuration, the YARN and HDFS master components run on the same
-machine.  This is useful for lower-resource deployments::
-
-    juju deploy apache-hadoop-hdfs-master hdfs-master
-    juju deploy apache-hadoop-hdfs-checkpoint secondary-namenode --to 1
-    juju deploy apache-hadoop-yarn-master yarn-master --to 1
-    juju deploy apache-hadoop-compute-slave compute-slave
-    juju deploy apache-hadoop-client client
-    juju add-relation yarn-master hdfs-master
-    juju add-relation secondary-namenode hdfs-master
-    juju add-relation compute-slave yarn-master
-    juju add-relation compute-slave hdfs-master
-    juju add-relation client yarn-master
-    juju add-relation client hdfs-master
-
-Note that the machine number (`--to 1`) should match the machine number
-for the `hdfs-master` charm.  If you previously deployed other services
-in your environment, you may need to adjust the machine number appropriately.
-
-
-### Scale Out Usage: Separate HDFS, YARN, and compute nodes
-
-In this configuration the HDFS and YARN deployments operate on
-different service units as separate services::
-
-    juju deploy apache-hadoop-hdfs-master hdfs-master
-    juju deploy apache-hadoop-hdfs-checkpoint secondary-namenode
-    juju deploy apache-hadoop-yarn-master yarn-master
-    juju deploy apache-hadoop-compute-slave compute-slave -n 3
-    juju deploy apache-hadoop-client client
-    juju add-relation yarn-master hdfs-master
-    juju add-relation secondary-namenode hdfs-master
-    juju add-relation compute-slave yarn-master
-    juju add-relation compute-slave hdfs-master
-    juju add-relation client yarn-master
-    juju add-relation client hdfs-master
-
-The `-n 3` option can be adjusted according to the number of compute nodes
-you need.  You can also add additional compute nodes later::
-
-    juju add-unit compute-slave -n 2
-
-
-### To deploy a Hadoop service with elasticsearch service::
-    # deploy ElasticSearch locally:
-    **juju deploy elasticsearch elasticsearch**
-    # elasticsearch-hadoop.jar file will be added to LIBJARS path
-    # Recommanded to use hadoop -libjars option to included elk jar file
-    **juju add-unit -n elasticsearch**
-    # deploy hive service by any senarios mentioned above
-    # associate Hive with elasticsearch
-    **juju add-relation {hadoop master}:elasticsearch elasticsearch:client**
+This charm is intended to be deployed via one of the
+[bundles](https://jujucharms.com/q/bigdata-dev/apache?type=bundle).
+For example:
+
+    juju quickstart u/bigdata-dev/apache-analytics-sql
+
+This will deploy the Apache Hadoop platform with Apache Hive available to
+perform SQL-like queries against your data.
+
+You can also manually load and run map-reduce jobs via the client:
+
+    juju scp my-job.jar client/0:
+    juju ssh client/0
+    hadoop jar my-job.jar
+
+
+### Scaling
+
+The compute-slave node is the "workhorse" of the Apache Hadoop platform.
+To scale your deployment's performance, you can simply add more compute-slave
+units.  For example, to add three mode units:
+
+    juju add-unit compute-slave -n 3
 
 
 ## Deploying in Network-Restricted Environments
@@ -121,12 +42,14 @@
 access. To deploy in this environment, you will need a local mirror to serve
 the packages and resources required by these charms.
 
+
 ### Mirroring Packages
 
 You can setup a local mirror for apt packages using squid-deb-proxy.
 For instructions on configuring juju to use this, see the
 [Juju Proxy Documentation](https://juju.ubuntu.com/docs/howto-proxies.html).
 
+
 ### Mirroring Resources
 
 In addition to apt packages, the Apache Hadoop charms require a few binary
@@ -144,14 +67,19 @@
 
 You can fetch the resources for all of the Apache Hadoop charms
 (`apache-hadoop-hdfs-master`, `apache-hadoop-yarn-master`,
-`apache-hadoop-compute-slave`, `apache-hadoop-client`, etc) into a single
+`apache-hadoop-hdfs-secondary`, `apache-hadoop-client`, etc) into a single
 directory and serve them all with a single `juju resources serve` instance.
 
 
 ## Contact Information
-amir sanjar <[e-mail]>
+
+* Amir Sanjar <[e-mail]>
+* Cory Johns <[e-mail]>
+* Kevin Monroe <[e-mail]>
+
 
 ## Hadoop
+
 - [Apache Hadoop](http://hadoop.apache.org/) home page
 - [Apache Hadoop bug trackers](http://hadoop.apache.org/issue_tracking.html)
 - [Apache Hadoop mailing lists](http://hadoop.apache.org/mailing_lists.html)

=== modified file 'resources.yaml'
--- resources.yaml	2015-03-06 22:28:48 +0000
+++ resources.yaml	2015-03-11 18:02:10 +0000
@@ -8,8 +8,8 @@
   six:
     pypi: six
   charmhelpers:
-    pypi: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150306222841-0ibvrfaungfdtkn8/charmhelpers0.2.2.ta-20150304033309-4fa7ewnosqavnwms-1/charmhelpers-0.2.2.tar.gz
-    hash: 787fc1cc70fc89e653b08dd192ec702d844709e450ff67577b7c5e99b6bbf39b
+    pypi: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150310214330-f2bk32gk92iinrx8/charmhelpers0.2.2.ta-20150304033309-4fa7ewnosqavnwms-1/charmhelpers-0.2.2.tar.gz
+    hash: a1cafa5e315d3a33db15a8e18f56b4c64d47c2c1c6fcbdba81e42bb00642971c
     hash_type: sha256
 optional_resources:
   hadoop-aarch64: