Juju Charms Collection
logstash-indexer package

Merge lp:~paulcz/charms/precise/logstash-indexer/trunk into lp:charms/logstash-indexer

  1. Precise Pangolin (12.04)
  2. trunk
  3. Merge into trunk
Proposed by Paul Czarkowski
Status: Merged
Merged at revision: 7
Proposed branch: lp:~paulcz/charms/precise/logstash-indexer/trunk
Merge into: lp:charms/logstash-indexer
Diff against target: 276 lines (+133/-32)
9 files modified
README.md (+20/-20)
files/charm/50-filters.conf (+40/-0)
files/charm/logstash-indexer.conf (+1/-1)
hooks/amqp-relation-changed (+43/-0)
hooks/amqp-relation-joined (+10/-0)
hooks/common.sh (+5/-3)
hooks/install (+8/-6)
metadata.yaml (+5/-1)
revision (+1/-1)
To merge this branch: bzr merge lp:~paulcz/charms/precise/logstash-indexer/trunk
Reviewer Review Type Date Requested Status
Marco Ceppi (community) Approve
Review via email: mp+188244@code.launchpad.net

Description of the change

* Updated to support logstash 1.2.x

To post a comment you must log in.
lp:~paulcz/charms/precise/logstash-indexer/trunk updated
8. By Paul Czarkowski

amqp relationship

9. By Paul Czarkowski

filters

10. By Paul Czarkowski

remove dropping message

Revision history for this message
Marco Ceppi (marcoceppi) wrote :

LGTM +1

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'README.md'
--- README.md 2012-11-18 22:27:48 +0000
+++ README.md 2013-10-18 15:38:52 +0000
@@ -19,37 +19,37 @@
19example 1 - Just the Indexer + Kibana19example 1 - Just the Indexer + Kibana
20=====================================20=====================================
2121
22juju deploy --repository=logstash local:precise/logstash-indexer22 juju deploy --repository=logstash local:precise/logstash-indexer
23juju deploy --repository=logstash local:precise/kibana23 juju deploy --repository=logstash local:precise/kibana
24juju add-relation kibana logstash-indexer:rest24 juju add-relation kibana logstash-indexer:rest
25juju expose kibana25 juju expose kibana
2626
27http://ip-of-kibana27http://ip-of-kibana
2828
29example 2 - Indexer + 2 x ElasticSearch + Kibana29example 2 - Indexer + 2 x ElasticSearch + Kibana
30============================================30============================================
3131
32juju deploy --repository=logstash local:precise/elasticsearch32 juju deploy --repository=logstash local:precise/elasticsearch
33juju add-unit elasticsearch33 juju add-unit elasticsearch
34juju deploy --repository=logstash local:precise/logstash-indexer34 juju deploy --repository=logstash local:precise/logstash-indexer
35juju add-relation logstash-indexer elasticsearch:cluster35 juju add-relation logstash-indexer elasticsearch:cluster
36juju deploy --repository=logstash local:precise/kibana36 juju deploy --repository=logstash local:precise/kibana
37juju add-relation kibana elasticsearch:rest37 juju add-relation kibana elasticsearch:rest
38juju expose kibana38 juju expose kibana
3939
40http://ip-of-kibana40http://ip-of-kibana
4141
42example 3 - Agent + Indexer + 2 x ElasticSearch + Kibana42example 3 - Agent + Indexer + 2 x ElasticSearch + Kibana
43=============================================43=============================================
4444
45juju deploy --repository=logstash local:precise/elasticsearch45 juju deploy --repository=logstash local:precise/elasticsearch
46juju add-unit elasticsearch46 juju add-unit elasticsearch
47juju deploy --repository=logstash local:precise/logstash-indexer47 juju deploy --repository=logstash local:precise/logstash-indexer
48juju add-relation logstash-indexer elasticsearch:cluster48 juju add-relation elasticsearch:cluster logstash-indexer
49juju deploy --repository=logstash local:precise/kibana49 juju deploy --repository=logstash local:precise/kibana
50juju add-relation kibana elasticsearch:rest50 juju add-relation elasticsearch:rest kibana
51juju expose kibana51 juju expose kibana
52juju deploy --repository=logstash local:precise/logstash-agent52 juju deploy --repository=logstash local:precise/logstash-agent
53juju add-relation logstash-agent logstash-indexer:input53 juju add-relation logstash-agent logstash-indexer:input
5454
55http://ip-of-kibana55http://ip-of-kibana
56\ No newline at end of file56\ No newline at end of file
5757
=== added file 'files/charm/50-filters.conf'
--- files/charm/50-filters.conf 1970-01-01 00:00:00 +0000
+++ files/charm/50-filters.conf 2013-10-18 15:38:52 +0000
@@ -0,0 +1,40 @@
1filter {
2
3 if [type] == "sensu" {
4 date {
5 match => ["timestamp", "ISO8601"]
6 }
7 mutate {
8 remove_field => [ "timestamp" ]
9 }
10 }
11
12 if [type] == "web-access" {
13 grok {
14 match => ["message", "%{COMBINEDAPACHELOG}"]
15 }
16 date {
17 match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z", "dd/MMM/yyyy:HH:mm:ss" ]
18 }
19 geoip {
20 source => "clientip"
21 }
22 mutate {
23 remove_field => [ "timestamp" ]
24 }
25 }
26
27 if [type] == "syslog" {
28 grok {
29 overwrite => "message"
30 match => {
31 "message" => "%{SYSLOGTIMESTAMP:timestamp} %{IPORHOST:host} (?:%{PROG:program}(?:\[%{POSINT:pid}\])?: )?%{GREEDYDATA:message}"
32 }
33 }
34 syslog_pri { }
35 date {
36 match => [ "timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
37 }
38 }
39
40}
0\ No newline at end of file41\ No newline at end of file
142
=== modified file 'files/charm/logstash-indexer.conf'
--- files/charm/logstash-indexer.conf 2012-11-18 22:27:48 +0000
+++ files/charm/logstash-indexer.conf 2013-10-18 15:38:52 +0000
@@ -18,5 +18,5 @@
18console log18console log
1919
20script20script
21 java -jar BASEPATH/bin/logstash-monolithic.jar agent -f BASEPATH/conf.d/ --log BASEPATH/logstash-indexer.log -v21 java -jar BASEPATH/bin/logstash.jar agent -f BASEPATH/conf.d/ --log BASEPATH/logstash-indexer.log
22end script22end script
23\ No newline at end of file23\ No newline at end of file
2424
=== added file 'hooks/amqp-relation-changed'
--- hooks/amqp-relation-changed 1970-01-01 00:00:00 +0000
+++ hooks/amqp-relation-changed 2013-10-18 15:38:52 +0000
@@ -0,0 +1,43 @@
1#!/bin/bash
2
3set -eux # -x for verbose logging to juju debug-log
4HOME=`dirname $0`
5. $HOME/common.sh
6
7juju-log "logstash_agent - Relationship Advice -> $JUJU_REMOTE_UNIT"
8juju-log Relation settings:
9relation-get
10juju-log Relation members:
11relation-list
12
13RABBIT_HOST=$(relation-get private-address)
14RABBIT_PASS=$(relation-get password)
15
16if [[ -z $RABBIT_HOST ]] || [[ -z $RABBIT_PASS ]] ; then
17 juju-log "logstash_agent - amqp_changed: rabbit_host||rabbit_password not set."
18 exit 0
19fi
20
21
22clustered=$(relation-get clustered)
23if [[ -n "$clustered" ]] ; then
24 juju-log "logstash_agent - ampq_changed: Configuring for access to haclustered rabbitmq service."
25 RABBIT_VIP=$(relation-get vip)
26 [[ -z "$vip" ]] && juju-log "logstash_agent - amqp_changed: Clustered but no vip." && exit 0
27 RABBIT_HOST="$RABBIT_VIP"
28fi
29
30
31cat > ${BASEPATH}/conf.d/input_rabbit.conf << EOF
32input {
33 rabbitmq {
34 exchange => "logstash"
35 host => "${RABBIT_HOST}"
36 user => "${RABBIT_USER}"
37 password => "${RABBIT_PASS}"
38 vhost => "${RABBIT_VHOST}"
39 }
40}
41EOF
42
43[ "$(initctl status logstash-indexer | grep running)" ] && initctl restart logstash-indexer || initctl start logstash-indexer
044
=== added file 'hooks/amqp-relation-joined'
--- hooks/amqp-relation-joined 1970-01-01 00:00:00 +0000
+++ hooks/amqp-relation-joined 2013-10-18 15:38:52 +0000
@@ -0,0 +1,10 @@
1#!/bin/bash
2
3set -eux # -x for verbose logging to juju debug-log
4HOME=`dirname $0`
5. $HOME/common.sh
6
7relation-set username=${RABBIT_USER}
8relation-set vhost=${RABBIT_VHOST}
9
10juju-log "logstash-indexer - amqp_joined: requesting credentials for $RABBIT_USER"
0\ No newline at end of file11\ No newline at end of file
112
=== modified file 'hooks/common.sh'
--- hooks/common.sh 2012-11-18 22:27:48 +0000
+++ hooks/common.sh 2013-10-18 15:38:52 +0000
@@ -3,13 +3,15 @@
3#set -eux # -x for verbose logging to juju debug-log3#set -eux # -x for verbose logging to juju debug-log
44
5# Set CONFIG constants5# Set CONFIG constants
6VERSION="1.1.4"6VERSION="1.2.1"
7BASEURL="https://logstash.objects.dreamhost.com/release"7BASEURL="https://download.elasticsearch.org/logstash/logstash"
8CHECKSUM="68ac6c953aad026752f69ae7c15b13f4"8CHECKSUM="863272192b52bccf1fc2cf839a888eaf"
9BASEPATH="/opt/logstash"9BASEPATH="/opt/logstash"
10USER="logstash"10USER="logstash"
11HOST=`unit-get private-address`11HOST=`unit-get private-address`
12CLUSTER_NAME="elasticsearch"12CLUSTER_NAME="elasticsearch"
13RABBIT_USER="logstash"
14RABBIT_VHOST="logstash"
1315
14# Grab CONFIG variables16# Grab CONFIG variables
15# Example :17# Example :
1618
=== modified file 'hooks/install'
--- hooks/install 2012-11-18 22:27:48 +0000
+++ hooks/install 2013-10-18 15:38:52 +0000
@@ -7,7 +7,7 @@
7juju-log "Installing required Packages via apt-get - java"7juju-log "Installing required Packages via apt-get - java"
8# was giving errors trying to install jre without update first.8# was giving errors trying to install jre without update first.
9apt-get update9apt-get update
10apt-get -y install default-jre-headless curl redis-server10apt-get -y install openjdk-7-jre-headless curl redis-server
11# set redis listener11# set redis listener
12sed -i.bak -e "s/bind 127.0.0.1/#bind ${HOST}/" /etc/redis/redis.conf12sed -i.bak -e "s/bind 127.0.0.1/#bind ${HOST}/" /etc/redis/redis.conf
13juju-log "Install upstart init script"13juju-log "Install upstart init script"
@@ -21,14 +21,14 @@
21mkdir -p ${BASEPATH}/data21mkdir -p ${BASEPATH}/data
22juju-log "install logstash and create config files"22juju-log "install logstash and create config files"
23# Check if a local copy of the jar exists23# Check if a local copy of the jar exists
24if [ -f ${HOME}/../files/charm/logstash-${VERSION}-monolithic.jar ]24if [ -f ${HOME}/../files/charm/logstash-${VERSION}-flatjar.jar ] logstash-1.2.1-flatjar.jar
25then25then
26 install -o root -g root -m 0644 ${HOME}/../files/charm/logstash-${VERSION}-monolithic.jar ${BASEPATH}/bin/logstash-monolithic.jar26 install -o root -g root -m 0644 ${HOME}/../files/charm/logstash-${VERSION}.jar ${BASEPATH}/bin/logstash.jar
27else27else
28 # Download logstash monolithic jar and checksum28 # Download logstash monolithic jar and checksum
29 wget -q -O ${BASEPATH}/bin/logstash-monolithic.jar ${BASEURL}/logstash-${VERSION}-monolithic.jar29 wget -q -O ${BASEPATH}/bin/logstash.jar ${BASEURL}/logstash-${VERSION}-flatjar.jar
30 if [[ $(md5sum "${BASEPATH}/bin/logstash-monolithic.jar" | cut -d " " -f 1) != ${CHECKSUM} ]]; then30 if [[ $(md5sum "${BASEPATH}/bin/logstash.jar" | cut -d " " -f 1) != ${CHECKSUM} ]]; then
31 juju-log --log-level CRITICAL "Checksum of ${BASEPATH}/bin/logstash-monolithic.jar failed"31 juju-log --log-level CRITICAL "Checksum of ${BASEPATH}/bin/logstash.jar failed"
32 exit 132 exit 1
33 fi33 fi
34fi34fi
@@ -36,6 +36,8 @@
36echo 'input { redis { host => "127.0.0.1" data_type => "list" type => "redis" tags => ["redis"] key => "logstash" message_format => "json_event" } } ' > ${BASEPATH}/conf.d/input-redis.conf36echo 'input { redis { host => "127.0.0.1" data_type => "list" type => "redis" tags => ["redis"] key => "logstash" message_format => "json_event" } } ' > ${BASEPATH}/conf.d/input-redis.conf
37echo 'output { elasticsearch { embedded => true } }' > ${BASEPATH}/conf.d/output-elasticsearch.conf37echo 'output { elasticsearch { embedded => true } }' > ${BASEPATH}/conf.d/output-elasticsearch.conf
3838
39install -o root -g root -m 0644 ${HOME}/../files/charm/50-filters.conf ${BASEPATH}/conf.d/50-filters.conf
40
39juju-log "create log file and pwn it"41juju-log "create log file and pwn it"
40touch /var/log/logstash-indexer.log42touch /var/log/logstash-indexer.log
41chown ${USER}. /var/log/logstash-indexer.log43chown ${USER}. /var/log/logstash-indexer.log
4244
=== modified file 'metadata.yaml'
--- metadata.yaml 2012-11-18 22:27:48 +0000
+++ metadata.yaml 2013-10-18 15:38:52 +0000
@@ -4,10 +4,14 @@
4description: |4description: |
5 Installs the logstash indexer. Initially stand alone version, 5 Installs the logstash indexer. Initially stand alone version,
6 but would like to add charms and hooks for ElasticSearch, 6 but would like to add charms and hooks for ElasticSearch,
7 Kibana, and other stuffs. 7 Kibana, and other stuffs.
8categories:
9 - applications
8requires:10requires:
9 cluster:11 cluster:
10 interface: elasticsearch12 interface: elasticsearch
13 amqp:
14 interface: rabbitmq
11provides:15provides:
12 input:16 input:
13 interface: redis17 interface: redis
1418
=== modified file 'revision'
--- revision 2012-11-01 04:07:22 +0000
+++ revision 2013-10-18 15:38:52 +0000
@@ -1,1 +1,1 @@
1011

Subscribers

People subscribed via source and target branches

to all changes: