Merge lp:~canonical-is-sa/charms/trusty/kibana/kibana4 into lp:charms/trusty/kibana

[Michael Nelson]

Fixes nginx proxy so only local requests to :9200 are forwarded to elasticsearch units.
Updates to latest kibana4 as default.
Specifies missing kibana config file and uses explicit logging option.
Adds support for exec.d pre-install hook.

Note: I wasn't planning on merging these changes to upstream kibana, because I thought you might see fallout if the kibana charm now defaults to kibana4, but other related charms (elatsicsearch, logstash) don't default to the required newer versions. But since the kibana4 changes have already been landed, here are some small changes :)

I'll comment below inline on each change, but I've tested the 11-scale-elastec locally:

dev-trusty# ~/charms/kibana/trusty/kibana
$ ./tests/11-scale-elastic.py
2015-11-09 03:09:35 Starting deployment of local
2015-11-09 03:09:35 Deploying services...
2015-11-09 03:09:35 Deploying service elasticsearch using cs:trusty/elasticsearch-12
2015-11-09 03:09:43 Deploying service kibana using /home/michael/charms/kibana/trusty/kibana
2015-11-09 03:09:48 Deploying service logstash-indexer using cs:precise/logstash-indexer-4
2015-11-09 03:13:08 Adding relations...
2015-11-09 03:13:08 Adding relation logstash-indexer:cluster <-> elasticsearch:client
2015-11-09 03:13:09 Adding relation kibana:rest <-> elasticsearch:client
2015-11-09 03:14:09 Exposing service 'kibana'
2015-11-09 03:14:09 Deployment complete in 275.00 seconds
dev-trusty# ~/charms/kibana/trusty/kibana
$

[Michael Nelson (michael.nelson)]

Adding inline comments.

[Review Queue (review-queue)]

This item has failed automated testing! Results available here http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/1423/

review: Needs Fixing (automated testing)

[Review Queue (review-queue)]

The results (PASS) are in and available here: http://juju-ci.vapour.ws:8080/job/charm-bundle-test-aws/1420/

review: Approve (automated testing)

[Chris MacNaughton (chris.macnaughton)]

Hi Michael,

Great work on these changes, making things more secure for everybody!

Looks good to me, according to http://reports.vapour.ws/all-bundle-and-charm-results/https%3A%252F%252Fcode.launchpad.net%252F~canonical-is-sa%252Fcharms%252Ftrusty%252Fkibana%252Fkibana4%252F%2Bmerge%252F276944 tests are in fact all passing.

Approve.

review: Approve

[Konstantinos Tsakalozos (kos.tsakalozos)]

LGTM. Thank you for doing this!

review: Approve

[Kevin W Monroe (kwmonroe)]

Thanks for these changes and the reviews! I made a small tweak to the README.md, which was still referencing the removed unpack dir:

-checksum verification, but respecting the 'kibana_source_unpack_directory'.
+checksum verification, and unpacked to /srv/kibana4.

This merge has been committed to lp:charms/trusty/kibana. An updated charm should be available in the store shortly. Thanks again!

review: Approve

=== modified file 'README.md'
--- README.md	2015-09-30 15:40:43 +0000
+++ README.md	2015-11-09 03:21:53 +0000
@@ -31,7 +31,7 @@
 configured values in 'kibana_source' and 'kibana_source_checksum', which
 default to::
 
-  https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz
+  https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz
 
 It will, obviously, require internet access on deployed nodes.
 

=== modified file 'config.yaml'
--- config.yaml	2015-09-30 15:40:43 +0000
+++ config.yaml	2015-11-09 03:21:53 +0000
@@ -1,15 +1,11 @@
 options:
     kibana_source:
         type: string
-        default: 'https://download.elastic.co/kibana/kibana/kibana-4.1.2-linux-x64.tar.gz'
+        default: 'https://download.elastic.co/kibana/kibana/kibana-4.2.0-linux-x64.tar.gz'
         description: URL to download the source for kibana
-    kibana_source_unpack_directory:
-        type: string
-        default: 'kibana-4.1.2-linux-x64'
-        description: the name of the directory the tar.gz will unpack to
     kibana_source_checksum:
         type: string
-        default: '45e67114f7dac4ccac8118bf98ee8f6362c7a6a1'
+        default: '094e197c687012211ae0e291faf3111d47e714d9'
         description: the sha1sum of the kibana_source
     port:
         type: int

=== modified file 'files/charm/config.yml'
--- files/charm/config.yml	2015-10-02 17:04:38 +0000
+++ files/charm/config.yml	2015-11-09 03:21:53 +0000
@@ -5,7 +5,8 @@
 host: "0.0.0.0"
 
 # The Elasticsearch instance to use for all your queries.
-elasticsearch_url: "http://localhost:9200"
+# This is proxied via /etc/nginx/sites-enabled/kibana_load_balancer
+elasticsearch_url: http://localhost:9200
 
 # preserve_elasticsearch_host true will send the hostname specified in `elasticsearch`. If you set it to false,
 # then the host you use to connect to *this* Kibana instance will be sent.

=== modified file 'files/charm/kibana.init'
--- files/charm/kibana.init	2015-10-02 17:04:38 +0000
+++ files/charm/kibana.init	2015-11-09 03:21:53 +0000
@@ -28,5 +28,5 @@
   export PATH HOME
   test -n "${JAVACMD}" && export JAVACMD
 
-  exec nice -n ${KB_NICE} ${KB_HOME}/bin/kibana 2>&1 ${KB_LOG_FILE}
+  exec nice -n ${KB_NICE} ${KB_HOME}/bin/kibana --config /etc/kibana/config.yml --log-file ${KB_LOG_FILE}
 end script

=== modified file 'files/charm/kibana_lb.conf'
--- files/charm/kibana_lb.conf	2015-10-09 20:54:45 +0000
+++ files/charm/kibana_lb.conf	2015-11-09 03:21:53 +0000
@@ -6,7 +6,10 @@
   listen *:9200;
 
   location / {
+    # Only proxy local requests for ES.
+    allow 127.0.0.1;
+    deny all;
     proxy_pass http://es_cluster;
     proxy_read_timeout 90;
   }
-}
\ No newline at end of file
+}

=== modified file 'hooks/config-changed'
--- hooks/config-changed	2015-10-09 20:54:45 +0000
+++ hooks/config-changed	2015-11-09 03:21:53 +0000
@@ -6,8 +6,6 @@
 
 sed -i "s/listen.*/listen *:$PORT;/" /etc/nginx/sites-available/kibana
 
-# sed -i "s@elasticsearch: .*@elasticsearch: '//'+window.location.hostname+'$KIBANA_PATH',@" /srv/kibana4/config.js
-
 service nginx reload
 service kibana restart
 

=== modified file 'hooks/install'
--- hooks/install	2015-10-02 17:04:38 +0000
+++ hooks/install	2015-11-09 03:21:53 +0000
@@ -1,9 +1,12 @@
 #!/bin/bash
 set -eux
+
+juju-log 'Invoking charm-pre-install hooks'
+[ -d exec.d ] && ( for f in exec.d/*/charm-pre-install; do [ -x $f ] && /bin/sh -c "$f"; done )
+
 HOME=$PWD
 KIBANA_TARBALL=files/kibana.tgz
 KIBANA_SOURCE=$(config-get kibana_source)
-KIBANA_SOURCE_UNPACK_DIRECTORY=$(config-get kibana_source_unpack_directory)
 KIBANA_SOURCE_CHECKSUM=$(config-get kibana_source_checksum)
 HOST=$(unit-get private-address)
 
@@ -30,20 +33,21 @@
 install -o root -g root -m 0644 files/charm/nginx_lb.conf /etc/nginx/sites-available/es_cluster
 install -o root -g root -m 0644 files/charm/kibana_lb.conf /etc/nginx/sites-available/kibana_load_balancer
 install -o root -g root -m 0644 files/charm/kibana.init /etc/init/kibana.conf
+mkdir -p /etc/kibana/
+install -o root -g root -m 0644 files/charm/config.yml /etc/kibana/config.yml
 pushd /srv
 
 if [[ ! -d kibana4 ]]; then
     tar xzvf /tmp/kibana.tgz
-    mv ${KIBANA_SOURCE_UNPACK_DIRECTORY} kibana4
+    # Note: charm does not currently support changing kibana source.
+    mv kibana-4.* kibana4
+    chown -R kibana:kibana kibana4
 fi
 
 rm -f /etc/nginx/sites-enabled/default
 
 popd
 
-# rm -f /srv/kibana4/config.js
-# install -o root -g root -m 0644 files/charm/config.js /srv/kibana4/config.js
-
 hooks/config-changed
 
 ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/kibana
@@ -51,4 +55,4 @@
 ln -s /etc/nginx/sites-available/kibana_load_balancer /etc/nginx/sites-enabled/kibana_load_balancer
 
 service nginx restart
-service kibana restart
\ No newline at end of file
+service kibana restart

=== added file 'hooks/web-relation-joined'
--- hooks/web-relation-joined	1970-01-01 00:00:00 +0000
+++ hooks/web-relation-joined	2015-11-09 03:21:53 +0000
@@ -0,0 +1,9 @@
+#!/bin/bash
+set -eux
+juju-log $JUJU_REMOTE_UNIT joined
+
+PORT=$(config-get port)
+
+# Set ip/port for web relation
+relation-set port="$PORT"
+relation-set host=`unit-get private-address`

=== removed file 'hooks/web-relation-joined'
--- hooks/web-relation-joined	2015-04-01 20:07:40 +0000
+++ hooks/web-relation-joined	1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-#!/bin/bash
-set -eux
-juju-log $JUJU_REMOTE_UNIT joined
-
-PORT=$(config-get port)
-
-# Set ip/port for web relation
-relation-set port="$PORT"
-relation-set host=`unit-get private-address`

=== modified file 'tests/11-scale-elastic.py'
--- tests/11-scale-elastic.py	2015-11-04 21:49:41 +0000
+++ tests/11-scale-elastic.py	2015-11-09 03:21:53 +0000
@@ -50,26 +50,26 @@
 
     time.sleep(3)
 
-    url = 'http://%s:9200/_all/_search' % kibana_unit.info['public-address']
-    test_dir = os.path.dirname(__file__)
-    test_data_file = os.path.join(test_dir, '11-scale-elastic-query.json')
-    with open(test_data_file) as fp:
-        test_data = json.load(fp)
-    r = requests.get(url, params=test_data)
-    if not r.ok:
-        amulet.raise_status(amulet.FAIL,
-                            msg="Error performing search.")
+    # Ensure the kibana unit can query ES via localhost (it proxies local 9200
+    # queries to ES)
+    url = 'http://127.0.0.1:9200/_all/_search'
+    result = kibana_unit.run(
+        "curl -XGET {url} -d '@11-scale-elastic-query.json'".format(url=url))
+    if not result[1] == 0:
+        amulet.raise_status(
+            amulet.FAIL, msg="Error performing search. Error code: {}. "
+                             "Result:\n{}".format(result[1], result[0]))
     try:
-        result = r.json()
+        result = json.loads(result[0])
         if result['hits']['total'] != 1:
             amulet.raise_status(amulet.FAIL,
                                 msg="Error inserting value in elasticsearch.")
     except ValueError:
         exc_type, value, traceback = sys.exc_info()
         exc_text = "Exception: (%s) %s. %s" % (value, exc_type, traceback)
-        amulet.raise_status(amulet.FAIL,
-                            msg="Value returned not a valid json." + r.text +
-                            exc_text)
+        amulet.raise_status(
+            amulet.FAIL, msg="Value returned not a valid json.\n{}\n{}".format(
+                result[0], exc_text))
 
 
 test_web_interface()