Merge lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk into lp:charms/trusty/apache-hadoop-plugin

[Kevin W Monroe]

[lp:~bigdata-dev/charms/trusty/apache-hadoop-plugin/trunk]

115. By Cory Johns on 2015-08-21

Fixed permissions on test_dist_config.py

[Kevin W Monroe (kwmonroe)]

Realtime syslog analytics bundle test looked good. Merged.

review: Approve

=== modified file 'DEV-README.md'
--- DEV-README.md	2015-06-29 14:23:33 +0000
+++ DEV-README.md	2015-08-21 21:51:35 +0000
@@ -27,11 +27,19 @@
 
 Additionally, the `JAVA_HOME`, `HADOOP_HOME`, `HADOOP_CONF_DIR`, and other
 environment variables will be set via `/etc/environment`.  This includes putting
+<<<<<<< TREE
 the Hadoop bin and sbin directories on the `PATH`.  There are
 [helpers](http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/files/head:/common/noarch/)
 in `charmhelpers.contrib.bigdata.utils` to assist with using the environment
 file. For example, to run the `hdfs` command to create a directory as the
 `ubuntu` user:
+=======
+the Hadoop bin and sbin directories on the `PATH`.  There are
+[helpers](https://git.launchpad.net/bigdata-data/tree/common/noarch)
+in `charmhelpers.contrib.bigdata.utils` to assist with using the environment
+file. For example, to run the `hdfs` command to create a directory as the
+`ubuntu` user:
+>>>>>>> MERGE-SOURCE
 
     from charmhelpers.contrib.bigdata.utils import run_as
     run_as('ubuntu', 'hdfs', 'dfs', '-mkdir', '-p', '/home/ubuntu/foo')
@@ -102,10 +110,17 @@
 
 ## Manual Deployment
 
+<<<<<<< TREE
 The easiest way to deploy the core Apache Hadoop platform is to use one of
 the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
 However, to manually deploy the base Apache Hadoop platform without using one
 of the bundles, you can use the following:
+=======
+The easiest way to deploy an Apache Hadoop platform is to use one of
+the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
+However, to manually deploy the base Apache Hadoop platform without using one
+of the bundles, you can use the following:
+>>>>>>> MERGE-SOURCE
 
     juju deploy apache-hadoop-hdfs-master hdfs-master
     juju deploy apache-hadoop-hdfs-secondary secondary-namenode

=== modified file 'README.md'
--- README.md	2015-06-18 17:12:11 +0000
+++ README.md	2015-08-21 21:51:35 +0000
@@ -22,7 +22,11 @@
 If you wanted to also wanted to be able to analyze your data using Apache Pig,
 you could deploy it and attach it to the same plugin:
 
+<<<<<<< TREE
     juju deploy cs:~bigdata-charmers/trusty/apache-pig pig
+=======
+    juju deploy apache-pig pig
+>>>>>>> MERGE-SOURCE
     juju add-relation plugin pig
 
 ## Benchmarking
@@ -92,17 +96,19 @@
 of these resources:
 
     sudo pip install jujuresources
-    juju resources fetch --all apache-hadoop-plugin/resources.yaml -d /tmp/resources
-    juju resources serve -d /tmp/resources
+    juju-resources fetch --all /path/to/resources.yaml -d /tmp/resources
+    juju-resources serve -d /tmp/resources
 
 This will fetch all of the resources needed by this charm and serve them via a
-simple HTTP server. You can then set the `resources_mirror` config option to
-have the charm use this server for retrieving resources.
+simple HTTP server. The output from `juju-resources serve` will give you a
+URL that you can set as the `resources_mirror` config option for this charm.
+Setting this option will cause all resources required by this charm to be
+downloaded from the configured URL.
 
 You can fetch the resources for all of the Apache Hadoop charms
 (`apache-hadoop-hdfs-master`, `apache-hadoop-yarn-master`,
 `apache-hadoop-compute-slave`, `apache-hadoop-plugin`, etc) into a single
-directory and serve them all with a single `juju resources serve` instance.
+directory and serve them all with a single `juju-resources serve` instance.
 
 
 ## Contact Information

=== modified file 'actions/parseTerasort.py'
--- actions/parseTerasort.py	2015-06-08 20:05:37 +0000
+++ actions/parseTerasort.py	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 """
 Simple script to parse cassandra-stress' transaction results
 and reformat them as JSON for sending back to juju

=== modified file 'actions/teragen'
--- actions/teragen	2015-06-08 21:32:27 +0000
+++ actions/teragen	2015-08-21 21:51:35 +0000
@@ -3,6 +3,10 @@
 SIZE=`action-get size`
 IN_DIR=`action-get indir`
 
+# The plugin uses a venv to avoid conflicting with system bits; ensure our
+# venv is activated.
+. ${CHARM_DIR}/.venv/bin/activate
+
 benchmark-start
 
 # NB: Escaped vars in the block below (e.g., \${HADOOP_HOME}) come from

=== modified file 'actions/terasort'
--- actions/terasort	2015-06-08 21:32:27 +0000
+++ actions/terasort	2015-08-21 21:51:35 +0000
@@ -25,6 +25,9 @@
 # from this outer scope
 su ubuntu << EOF
 . /etc/environment
+# The plugin uses a venv to avoid conflicting with system bits; ensure our
+# venv is activated.
+. ${CHARM_DIR}/.venv/bin/activate
 
 mkdir -p /opt/terasort/results/$run
 
@@ -45,4 +48,4 @@
 EOF
 PATH=$OLDPATH
 
-`cat /opt/terasort/results/$run/terasort.log | python $CHARM_DIR/actions/parseTerasort.py`
+`cat /opt/terasort/results/$run/terasort.log | $CHARM_DIR/.venv/bin/python $CHARM_DIR/actions/parseTerasort.py`

=== modified file 'dist.yaml'
--- dist.yaml	2015-04-16 15:45:57 +0000
+++ dist.yaml	2015-08-21 21:51:35 +0000
@@ -68,49 +68,3 @@
         owner: 'hdfs'
         group: 'hadoop'
         perms: 0775
-ports:
-    # Ports that need to be exposed, overridden, or manually specified.
-    # Only expose ports serving a UI or external API (i.e., namenode and
-    # resourcemanager).  Communication among units within the cluster does
-    # not need ports to be explicitly opened.
-    # If adding a port here, you will need to update
-    # charmhelpers.contrib.bigdata.handlers.apache or hooks/callbacks.py
-    # to ensure that it is supported.
-    namenode:
-        port: 8020
-        exposed_on: 'hdfs-master'
-    nn_webapp_http:
-        port: 50070
-        exposed_on: 'hdfs-master'
-    dn_webapp_http:
-        port: 50075
-        exposed_on: 'compute-slave'
-    resourcemanager:
-        port: 8032
-        exposed_on: 'yarn-master'
-    rm_webapp_http:
-        port: 8088
-        exposed_on: 'yarn-master'
-    rm_log:
-        port: 19888
-    nm_webapp_http:
-        port: 8042
-        exposed_on: 'compute-slave'
-    jobhistory:
-        port: 10020
-    jh_webapp_http:
-        port: 19888
-        exposed_on: 'yarn-master'
-    # TODO: support SSL
-    #nn_webapp_https:
-    #    port: 50470
-    #    exposed_on: 'hdfs-master'
-    #dn_webapp_https:
-    #    port: 50475
-    #    exposed_on: 'compute-slave'
-    #rm_webapp_https:
-    #    port: 8090
-    #    exposed_on: 'yarn-master'
-    #nm_webapp_https:
-    #    port: 8044
-    #    exposed_on: 'compute-slave'

=== modified file 'hooks/callbacks.py'
--- hooks/callbacks.py	2015-06-25 15:41:48 +0000
+++ hooks/callbacks.py	2015-08-21 21:51:35 +0000
@@ -24,28 +24,28 @@
 def update_blocked_status():
     if unitdata.kv().get('charm.active', False):
         return
-    rels = (
-        ('Yarn', 'ResourceManager', ResourceManager()),
+    rels = [
         ('HDFS', 'NameNode', NameNode()),
-    )
+    ]
     missing_rel = [rel for rel, res, impl in rels if not impl.connected_units()]
-    missing_hosts = [rel for rel, res, impl in rels if not impl.am_i_registered()]
-    not_ready = [(rel, res) for rel, res, impl in rels if not impl.is_ready()]
+    rels.append(('Yarn', 'ResourceManager', ResourceManager()))
+    not_ready = [(rel, res) for rel, res, impl in rels if impl.connected_units() and not impl.is_ready()]
+    missing_hosts = [rel for rel, res, impl in rels if impl.connected_units() and not impl.am_i_registered()]
     if missing_rel:
         hookenv.status_set('blocked', 'Waiting for relation to %s master%s' % (
             ' and '.join(missing_rel),
             's' if len(missing_rel) > 1 else '',
         )),
-    elif missing_hosts:
-        hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % (
-            ' and '.join(missing_hosts),
-        ))
     elif not_ready:
         unready_rels, unready_ress = zip(*not_ready)
         hookenv.status_set('waiting', 'Waiting for %s to provide %s' % (
             ' and '.join(unready_rels),
             ' and '.join(unready_ress),
         ))
+    elif missing_hosts:
+        hookenv.status_set('waiting', 'Waiting for /etc/hosts registration on %s' % (
+            ' and '.join(missing_hosts),
+        ))
 
 
 def update_working_status():
@@ -56,5 +56,18 @@
 
 
 def update_active_status():
-    unitdata.kv().set('charm.active', True)
-    hookenv.status_set('active', 'Ready')
+    hdfs_ready = NameNode().is_ready()
+    yarn_connected = ResourceManager().connected_units()
+    yarn_ready = ResourceManager().is_ready()
+    if hdfs_ready and (not yarn_connected or yarn_ready):
+        unitdata.kv().set('charm.active', True)
+        hookenv.status_set('active', 'Ready%s' % (
+            '' if yarn_ready else ' (HDFS only)'
+        ))
+    else:
+        clear_active_flag()
+        update_blocked_status()
+
+
+def clear_active_flag():
+    unitdata.kv().set('charm.active', False)

=== modified file 'hooks/common.py'
--- hooks/common.py	2015-06-25 15:41:48 +0000
+++ hooks/common.py	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
@@ -14,12 +14,21 @@
 Common implementation for all hooks.
 """
 
+import os
 import jujuresources
 from charmhelpers.core import hookenv
 from charmhelpers.core import unitdata
 from charmhelpers.core import charmframework
 
 
+# ensure that the venv is used for installing resources
+# (venv is used to ensure library versions needed by plugin
+# don't conflict with the charm the plugin is supporting)
+os.environ['PATH'] = os.pathsep.join([
+    os.path.join(hookenv.charm_dir(), '.venv/bin'),
+    os.environ['PATH']])
+
+
 def bootstrap_resources():
     """
     Install required resources defined in resources.yaml
@@ -51,7 +60,7 @@
 
     # list of keys required to be in the dist.yaml
     client_reqs = ['vendor', 'hadoop_version', 'packages', 'groups', 'users',
-                   'dirs', 'ports']
+                   'dirs']
     dist_config = jujubigdata.utils.DistConfig(filename='dist.yaml',
                                                required_keys=client_reqs)
     hadoop = jujubigdata.handlers.HadoopBase(dist_config)
@@ -71,26 +80,44 @@
             ],
         },
         {
-            'name': 'plugin',
+            'name': 'hdfs',
             'provides': [
                 jujubigdata.relations.HadoopPlugin(),
             ],
             'requires': [
                 hadoop.is_installed,
+                hdfs_relation,
+            ],
+            'callbacks': [
+                callbacks.update_working_status,
+                hdfs_relation.register_provided_hosts,
+                jujubigdata.utils.manage_etc_hosts,
+                hdfs.configure_client,
+                callbacks.update_active_status,
+            ],
+            'cleanup': [
+                callbacks.clear_active_flag,
+                callbacks.update_blocked_status,
+            ],
+        },
+        {
+            'name': 'yarn',
+            'provides': [],
+            'requires': [
+                hadoop.is_installed,
                 yarn_relation,
-                hdfs_relation,
             ],
             'callbacks': [
                 callbacks.update_working_status,
                 yarn_relation.register_provided_hosts,
-                hdfs_relation.register_provided_hosts,
                 jujubigdata.utils.manage_etc_hosts,
                 yarn.install_demo,
                 yarn.configure_client,
-                hdfs.configure_client,
                 callbacks.update_active_status,
             ],
-            'cleanup': [],
+            'cleanup': [
+                callbacks.update_blocked_status,
+            ],
         },
     ])
     manager.manage()

=== modified file 'hooks/config-changed'
--- hooks/config-changed	2015-02-09 18:13:28 +0000
+++ hooks/config-changed	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at

=== modified file 'hooks/hadoop-plugin-relation-changed'
--- hooks/hadoop-plugin-relation-changed	2015-04-28 13:38:46 +0000
+++ hooks/hadoop-plugin-relation-changed	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at

=== added file 'hooks/hadoop-plugin-relation-departed'
--- hooks/hadoop-plugin-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/hadoop-plugin-relation-departed	2015-08-21 21:51:35 +0000
@@ -0,0 +1,15 @@
+#!.venv/bin/python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import common
+common.manage()

=== modified file 'hooks/namenode-relation-changed'
--- hooks/namenode-relation-changed	2015-05-07 15:27:21 +0000
+++ hooks/namenode-relation-changed	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at

=== added file 'hooks/namenode-relation-departed'
--- hooks/namenode-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/namenode-relation-departed	2015-08-21 21:51:35 +0000
@@ -0,0 +1,16 @@
+#!.venv/bin/python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import common
+
+common.manage()

=== modified file 'hooks/resourcemanager-relation-changed'
--- hooks/resourcemanager-relation-changed	2015-05-07 15:11:29 +0000
+++ hooks/resourcemanager-relation-changed	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at

=== added file 'hooks/resourcemanager-relation-departed'
--- hooks/resourcemanager-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/resourcemanager-relation-departed	2015-08-21 21:51:35 +0000
@@ -0,0 +1,16 @@
+#!.venv/bin/python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import common
+
+common.manage()

=== modified file 'hooks/setup.py'
--- hooks/setup.py	2015-06-25 15:41:48 +0000
+++ hooks/setup.py	2015-08-21 21:51:35 +0000
@@ -9,6 +9,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import os
 import subprocess
 from glob import glob
 
@@ -17,12 +18,15 @@
     """
     Do any setup required before the install hook.
     """
-    install_pip()
+    setup_venv()
     install_bundled_resources()
 
 
-def install_pip():
-    subprocess.check_call(['apt-get', 'install', '-yq', 'python-pip', 'bzr'])
+def setup_venv():
+    if not os.path.exists('.venv/bin/python'):
+        subprocess.check_call(['apt-get', 'install', '-yq', 'python-virtualenv'])
+        subprocess.check_call(['virtualenv', '.venv'])
+    execfile('.venv/bin/activate_this.py', {'__file__': '.venv/bin/activate_this.py'})
 
 
 def install_bundled_resources():
@@ -30,4 +34,4 @@
     Install the bundled resources libraries.
     """
     archives = glob('resources/python/*')
-    subprocess.check_call(['pip', 'install'] + archives)
+    subprocess.check_call(['.venv/bin/pip', 'install'] + archives)

=== modified file 'hooks/start'
--- hooks/start	2015-02-09 18:13:28 +0000
+++ hooks/start	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at

=== modified file 'hooks/stop'
--- hooks/stop	2015-02-09 18:13:28 +0000
+++ hooks/stop	2015-08-21 21:51:35 +0000
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!.venv/bin/python
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at

=== modified file 'resources.yaml'
--- resources.yaml	2015-06-25 22:43:49 +0000
+++ resources.yaml	2015-08-21 21:51:35 +0000
@@ -4,7 +4,7 @@
   pathlib:
     pypi: path.py>=7.0
   jujubigdata:
-    pypi: jujubigdata>=2.0.2,<3.0.0
+    pypi: jujubigdata>=4.0.0,<5.0.0
   charm-benchmark:
     pypi: charm-benchmark>=1.0.1,<2.0.0
   java-installer:
@@ -12,19 +12,19 @@
     # If replaced with an alternate implementation, it must output *only* two
     # lines containing the JAVA_HOME path, and the Java version, respectively,
     # on stdout.  Upon error, it must exit with a non-zero exit code.
-    url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150625222410-qfck64q46ubx5i6p/javainstaller.sh-20150311213053-4vq7369jhlvc6qy8-1/java-installer.sh
-    hash: 8fdff60270ea4be7bbef1e013e503057fe6efc2f4b5761edebc206a54f303023
+    url: https://git.launchpad.net/bigdata-data/plain/common/noarch/java-installer.sh?id=baa0b74b86587f97b446f255deb96c8420021dd8
+    hash: f7df6937bdb4dcc60de559252b4e6b65c77959f871c7ef2e59af57832d7ddfca
     hash_type: sha256
 optional_resources:
   hadoop-aarch64:
-    url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150303192631-swhrf8f7q82si75t/hadoop2.4.1.tar.gz-20150303192554-7gqslr4m8ahkwiax-2/hadoop-2.4.1.tar.gz
+    url: https://git.launchpad.net/bigdata-data/plain/apache/aarch64/hadoop-2.4.1.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0
     hash: 03ad135835bfe413f85fe176259237a8
     hash_type: md5
   hadoop-ppc64le:
-    url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/kevin.monroe%40canonical.com-20150130165209-nuz1myezjpdx7eus/hadoop2.4.1ppc64le.t-20150130165148-s8i19s002ht88gio-2/hadoop-2.4.1-ppc64le.tar.gz
+    url: https://git.launchpad.net/bigdata-data/plain/apache/ppc64le/hadoop-2.4.1-ppc64le.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0
     hash: 09942b168a3db0d183b281477d3dae9deb7b7bc4b5783ba5cda3965b62e71bd5
     hash_type: sha256
   hadoop-x86_64:
-    url: http://bazaar.launchpad.net/~bigdata-dev/bigdata-data/trunk/download/cory.johns%40canonical.com-20150116154822-x5osw3zfhw6e03b1/hadoop2.4.1.tar.gz-20150116154748-yfa2j12rr5m53xd3-1/hadoop-2.4.1.tar.gz
+    url: https://git.launchpad.net/bigdata-data/plain/apache/x86_64/hadoop-2.4.1.tar.gz?id=c34a21c939f5fce9ab89b95d65fe2df50e7bbab0
     hash: a790d39baba3a597bd226042496764e0520c2336eedb28a1a3d5c48572d3b672
     hash_type: sha256

=== removed file 'resources/python/jujuresources-0.2.8.tar.gz'
Binary files resources/python/jujuresources-0.2.8.tar.gz	2015-06-25 15:41:48 +0000 and resources/python/jujuresources-0.2.8.tar.gz	1970-01-01 00:00:00 +0000 differ
=== added file 'resources/python/jujuresources-0.2.9.tar.gz'
Binary files resources/python/jujuresources-0.2.9.tar.gz	1970-01-01 00:00:00 +0000 and resources/python/jujuresources-0.2.9.tar.gz	2015-08-21 21:51:35 +0000 differ
=== modified file 'tests/remote/test_dist_config.py' (properties changed: -x to +x)