Merge lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-master/trunk into lp:charms/trusty/apache-hadoop-hdfs-master

[Kevin W Monroe]

[lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-master/trunk]

101. By Cory Johns on 2015-08-21

Fixed permissions on test_dist_config.py

[Kevin W Monroe (kwmonroe)]

Realtime syslog analytics bundle test looked good. Merged.

review: Approve

=== modified file 'DEV-README.md'
--- DEV-README.md	2015-06-18 17:07:45 +0000
+++ DEV-README.md	2015-08-21 21:51:24 +0000
@@ -61,10 +61,17 @@
 
 ## Manual Deployment
 
+<<<<<<< TREE
 The easiest way to deploy the core Apache Hadoop platform is to use one of
 the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
 However, to manually deploy the base Apache Hadoop platform without using one
 of the bundles, you can use the following:
+=======
+The easiest way to deploy an Apache Hadoop platform is to use one of
+the [apache bundles](https://jujucharms.com/u/bigdata-charmers/#bundles).
+However, to manually deploy the base Apache Hadoop platform without using one
+of the bundles, you can use the following:
+>>>>>>> MERGE-SOURCE
 
     juju deploy apache-hadoop-hdfs-master hdfs-master
     juju deploy apache-hadoop-hdfs-secondary secondary-namenode

=== modified file 'README.md'
--- README.md	2015-06-29 14:19:09 +0000
+++ README.md	2015-08-21 21:51:24 +0000
@@ -27,6 +27,35 @@
     hadoop jar my-job.jar
 
 
+## Status and Smoke Test
+
+The services provide extended status reporting to indicate when they are ready:
+
+    juju status --format=tabular
+
+This is particularly useful when combined with `watch` to track the on-going
+progress of the deployment:
+
+    watch -n 0.5 juju status --format=tabular
+
+The message for each unit will provide information about that unit's state.
+Once they all indicate that they are ready, you can perform a "smoke test"
+to verify that HDFS is working as expected using the built-in `smoke-test`
+action:
+
+    juju action do smoke-test
+
+After a few seconds or so, you can check the results of the smoke test:
+
+    juju action status
+
+You will see `status: completed` if the smoke test was successful, or
+`status: failed` if it was not.  You can get more information on why it failed
+via:
+
+    juju action fetch <action-id>
+
+
 ## Deploying in Network-Restricted Environments
 
 The Apache Hadoop charms can be deployed in environments with limited network
@@ -49,17 +78,19 @@
 of these resources:
 
     sudo pip install jujuresources
-    juju resources fetch --all apache-hadoop-compute-slave/resources.yaml -d /tmp/resources
-    juju resources serve -d /tmp/resources
+    juju-resources fetch --all /path/to/resources.yaml -d /tmp/resources
+    juju-resources serve -d /tmp/resources
 
 This will fetch all of the resources needed by this charm and serve them via a
-simple HTTP server. You can then set the `resources_mirror` config option to
-have the charm use this server for retrieving resources.
+simple HTTP server. The output from `juju-resources serve` will give you a
+URL that you can set as the `resources_mirror` config option for this charm.
+Setting this option will cause all resources required by this charm to be
+downloaded from the configured URL.
 
 You can fetch the resources for all of the Apache Hadoop charms
 (`apache-hadoop-hdfs-master`, `apache-hadoop-yarn-master`,
 `apache-hadoop-hdfs-secondary`, `apache-hadoop-plugin`, etc) into a single
-directory and serve them all with a single `juju resources serve` instance.
+directory and serve them all with a single `juju-resources serve` instance.
 
 
 ## Contact Information

=== modified file 'actions.yaml'
--- actions.yaml	2015-06-23 17:15:23 +0000
+++ actions.yaml	2015-08-21 21:51:24 +0000
@@ -4,3 +4,5 @@
     description: All of the HDFS processes can be stopped with this Juju action.
 restart-hdfs:
     description: All of the HDFS processes can be restarted with this Juju action.
+smoke-test:
+    description: Verify that HDFS is working by creating and removing a small file.

=== added file 'actions/smoke-test'
--- actions/smoke-test	1970-01-01 00:00:00 +0000
+++ actions/smoke-test	2015-08-21 21:51:24 +0000
@@ -0,0 +1,53 @@
+#!/usr/bin/env python
+
+import sys
+
+try:
+    from charmhelpers.core import hookenv
+    from charmhelpers.core import unitdata
+    from jujubigdata.utils import run_as
+    charm_ready = unitdata.kv().get('charm.active', False)
+except ImportError:
+    charm_ready = False
+
+if not charm_ready:
+    # might not have hookenv.action_fail available yet
+    from subprocess import call
+    call(['action-fail', 'HDFS service not yet ready'])
+
+
+# verify the hdfs-test directory does not already exist
+output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True)
+if '/tmp/hdfs-test' in output:
+    run_as('ubuntu', 'hdfs', 'dfs', '-rm', '-R', '/tmp/hdfs-test')
+    output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True)
+    if 'hdfs-test' in output:
+        hookenv.action_fail('Unable to remove existing hdfs-test directory')
+        sys.exit()
+
+# create the directory
+run_as('ubuntu', 'hdfs', 'dfs', '-mkdir', '-p', '/tmp/hdfs-test')
+run_as('ubuntu', 'hdfs', 'dfs', '-chmod', '-R', '777', '/tmp/hdfs-test')
+
+# verify the newly created hdfs-test subdirectory exists
+output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True)
+for line in output.split('\n'):
+    if '/tmp/hdfs-test' in line:
+        if 'ubuntu' not in line or 'drwxrwxrwx' not in line:
+            hookenv.action_fail('Permissions incorrect for hdfs-test directory')
+            sys.exit()
+        break
+else:
+    hookenv.action_fail('Unable to create hdfs-test directory')
+    sys.exit()
+
+# remove the directory
+run_as('ubuntu', 'hdfs', 'dfs', '-rm', '-R', '/tmp/hdfs-test')
+
+# verify the hdfs-test subdirectory has been removed
+output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True)
+if '/tmp/hdfs-test' in output:
+    hookenv.action_fail('Unable to remove hdfs-test directory')
+    sys.exit()
+
+hookenv.action_set({'outcome': 'success'})

=== modified file 'dist.yaml'
--- dist.yaml	2015-04-16 15:46:35 +0000
+++ dist.yaml	2015-08-21 21:51:24 +0000
@@ -73,44 +73,12 @@
     # Only expose ports serving a UI or external API (i.e., namenode and
     # resourcemanager).  Communication among units within the cluster does
     # not need ports to be explicitly opened.
-    # If adding a port here, you will need to update
-    # charmhelpers.contrib.bigdata.handlers.apache or hooks/callbacks.py
-    # to ensure that it is supported.
     namenode:
         port: 8020
-        exposed_on: 'hdfs-master'
     nn_webapp_http:
         port: 50070
         exposed_on: 'hdfs-master'
-    dn_webapp_http:
-        port: 50075
-        exposed_on: 'compute-slave'
-    resourcemanager:
-        port: 8032
-        exposed_on: 'yarn-master'
-    rm_webapp_http:
-        port: 8088
-        exposed_on: 'yarn-master'
-    rm_log:
-        port: 19888
-    nm_webapp_http:
-        port: 8042
-        exposed_on: 'compute-slave'
-    jobhistory:
-        port: 10020
-    jh_webapp_http:
-        port: 19888
-        exposed_on: 'yarn-master'
     # TODO: support SSL
     #nn_webapp_https:
     #    port: 50470
     #    exposed_on: 'hdfs-master'
-    #dn_webapp_https:
-    #    port: 50475
-    #    exposed_on: 'compute-slave'
-    #rm_webapp_https:
-    #    port: 8090
-    #    exposed_on: 'yarn-master'
-    #nm_webapp_https:
-    #    port: 8044
-    #    exposed_on: 'compute-slave'

=== modified file 'hooks/callbacks.py'
--- hooks/callbacks.py	2015-06-25 15:38:21 +0000
+++ hooks/callbacks.py	2015-08-21 21:51:24 +0000
@@ -37,3 +37,7 @@
         hookenv.status_set('waiting', 'Waiting for compute slaves to provide DataNodes')
     else:
         hookenv.status_set('blocked', 'Waiting for relation to compute slaves')
+
+
+def clear_active_flag():
+    unitdata.kv().set('charm.active', False)

=== modified file 'hooks/common.py'
--- hooks/common.py	2015-06-25 15:39:10 +0000
+++ hooks/common.py	2015-08-21 21:51:24 +0000
@@ -100,8 +100,10 @@
                 callbacks.update_active_status,
             ],
             'cleanup': [
+                callbacks.clear_active_flag,
                 charmframework.helpers.close_ports(dist_config.exposed_ports('hdfs-master')),
                 hdfs.stop_namenode,
+                callbacks.update_active_status,
             ],
         },
     ])

=== added file 'hooks/datanode-relation-departed'
--- hooks/datanode-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/datanode-relation-departed	2015-08-21 21:51:24 +0000
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+All hooks in this charm are managed by the Charm Framework.
+The framework helps manage dependencies and preconditions to ensure that
+steps are only executed when they can be successful.  As such, no additional
+code should be added to this hook; instead, please integrate new functionality
+into the 'callbacks' list in hooks/common.py.  New callbacks can be placed
+in hooks/callbacks.py, if necessary.
+
+See http://big-data-charm-helpers.readthedocs.org/en/latest/examples/framework.html
+for more information.
+"""
+import common
+common.manage()

=== added file 'hooks/namenode-relation-departed'
--- hooks/namenode-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/namenode-relation-departed	2015-08-21 21:51:24 +0000
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+All hooks in this charm are managed by the Charm Framework.
+The framework helps manage dependencies and preconditions to ensure that
+steps are only executed when they can be successful.  As such, no additional
+code should be added to this hook; instead, please integrate new functionality
+into the 'callbacks' list in hooks/common.py.  New callbacks can be placed
+in hooks/callbacks.py, if necessary.
+
+See http://big-data-charm-helpers.readthedocs.org/en/latest/examples/framework.html
+for more information.
+"""
+import common
+common.manage()

=== added file 'hooks/secondary-relation-departed'
--- hooks/secondary-relation-departed	1970-01-01 00:00:00 +0000
+++ hooks/secondary-relation-departed	2015-08-21 21:51:24 +0000
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+All hooks in this charm are managed by the Charm Framework.
+The framework helps manage dependencies and preconditions to ensure that
+steps are only executed when they can be successful.  As such, no additional
+code should be added to this hook; instead, please integrate new functionality
+into the 'callbacks' list in hooks/common.py.  New callbacks can be placed
+in hooks/callbacks.py, if necessary.
+
+See http://big-data-charm-helpers.readthedocs.org/en/latest/examples/framework.html
+for more information.
+"""
+import common
+common.manage()

=== modified file 'resources.yaml'
--- resources.yaml	2015-07-24 15:26:06 +0000
+++ resources.yaml	2015-08-21 21:51:24 +0000
@@ -4,7 +4,7 @@
   pathlib:
     pypi: path.py>=7.0
   jujubigdata:
-    pypi: jujubigdata>=2.0.2,<3.0.0
+    pypi: jujubigdata>=4.0.0,<5.0.0
   java-installer:
     # This points to a script which manages installing Java.
     # If replaced with an alternate implementation, it must output *only* two

=== added file 'resources/python/jujuresources-0.2.9.tar.gz'
Binary files resources/python/jujuresources-0.2.9.tar.gz	1970-01-01 00:00:00 +0000 and resources/python/jujuresources-0.2.9.tar.gz	2015-08-21 21:51:24 +0000 differ
=== renamed file 'resources/python/jujuresources-0.2.9.tar.gz' => 'resources/python/jujuresources-0.2.9.tar.gz.moved'
=== modified file 'tests/remote/test_dist_config.py' (properties changed: -x to +x)