Merge lp:~michael.nelson/charms/trusty/elasticsearch/add-ufw into lp:~charmers/charms/trusty/elasticsearch/trunk

[Michael Nelson]

Firewall admin port 9200 so it's only available on the localhost or for any juju-related clients.

Firewall admin port 9200 so it's only available on the localhost or for any juju-related clients.

I've tested this with one of my deployments. I'll do a followup branch which firewalls the node-to-node communication port (9300) to be available only to peers.

The trusty version of ansible (1.5.2) doesn't include the ufw module, so I'm including it in the charm here (we could also grab it from the network during install, but don't want to depend on 3rd party networks). It seems like a nice way to get newer ansible functionality (modules) for our trusty charms, without having to install ansible from a ppa or similar.

[Kapil Thangavelu (hazmat)]

ignoring expose and lack of open-port compatibility for now (es exposed publicly is bad juju). what happens on upgrade of an extant charm? afaics it will break them by installing ufw default deny mode without triggering reconsideration of the current clients. in general i've found this sort of add/one remove one pattern in response to relation events to be not ideal practice .. ie. just more complicated cause managing global state transitions and ordering, vs. just consider all the current ones and render the set / activate the delta ie a single entry point that all hooks can dispatch to.

[lp:~michael.nelson/charms/trusty/elasticsearch/add-ufw]

37. By Michael Nelson on 2014-07-30

Reset firewall for each new client.

38. By Michael Nelson on 2014-07-30

Set client_relation_id as a var for simpler tasks.

[Michael Nelson (michael.nelson)]

Thanks Kapil.

I've updated so that it resets the firewall on install, upgrade-charm
and client-relation-joined/departed.

I wasn't particularly happy having to reset the firewall each time -
it would have been much nicer to instead be writing a templated config
file for ufw (ie. only reloading the firewall if the config changes),
but afaict it's not possible.

I've included a demo of an upgrade-charm pre-ufw to this on the followup MP at:

https://code.launchpad.net/~michael.nelson/charms/trusty/elasticsearch/ufw-for-peers-too/+merge/225968

[Kapil Thangavelu (hazmat)]

lgtm

review: Approve

[Matt Bruzek (mbruzek)]

I have tested this and the follow up MP. I was able to deploy and the firewall rules looked to be correct in an ELK deployment.

+1

review: Approve

=== modified file 'README.md'
--- README.md	2014-07-28 18:10:43 +0000
+++ README.md	2014-07-30 06:36:30 +0000
@@ -33,7 +33,7 @@
 
 And when they have started you can inspect the cluster health:
 
-    juju ssh elasticsearch/0 "curl http://localhost:9200/_cat/health?v"
+    juju run --unit elasticsearch/0 "curl http://localhost:9200/_cat/health?v"
     epoch      timestamp cluster       status node.total node.data shards ...
     1404728290 10:18:10  elasticsearch green           2         2      0
 

=== added directory 'ansible_module_backports'
=== added file 'ansible_module_backports/ufw'
--- ansible_module_backports/ufw	1970-01-01 00:00:00 +0000
+++ ansible_module_backports/ufw	2014-07-30 06:36:30 +0000
@@ -0,0 +1,264 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+
+# (c) 2014, Ahti Kitsik <[e-mail]>
+# (c) 2014, Jarno Keskikangas <[e-mail]>
+# (c) 2013, Aleksey Ovcharenko <[e-mail]>
+# (c) 2013, James Martin <[e-mail]>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+DOCUMENTATION = '''
+---
+module: ufw
+short_description: Manage firewall with UFW
+description:
+    - Manage firewall with UFW.
+version_added: 1.6
+author: Aleksey Ovcharenko, Jarno Keskikangas, Ahti Kitsik
+notes:
+    - See C(man ufw) for more examples.
+requirements:
+    - C(ufw) package
+options:
+  state:
+    description:
+      - C(enabled) reloads firewall and enables firewall on boot.
+      - C(disabled) unloads firewall and disables firewall on boot.
+      - C(reloaded) reloads firewall.
+      - C(reset) disables and resets firewall to installation defaults.
+    required: false
+    choices: ['enabled', 'disabled', 'reloaded', 'reset']
+  policy:
+    description:
+      - Change the default policy for incoming or outgoing traffic.
+    required: false
+    alias: default
+    choices: ['allow', 'deny', 'reject']
+  direction:
+    description:
+      - Select direction for a rule or default policy command.
+    required: false
+    choices: ['in', 'out', 'incoming', 'outgoing']
+  logging:
+    description:
+      - Toggles logging. Logged packets use the LOG_KERN syslog facility.
+    choices: ['on', 'off', 'low', 'medium', 'high', 'full']
+    required: false
+  insert:
+    description:
+      - Insert the corresponding rule as rule number NUM
+    required: false
+  rule:
+    description:
+      - Add firewall rule
+    required: false
+    choices: ['allow', 'deny', 'reject', 'limit']
+  log:
+    description:
+      - Log new connections matched to this rule
+    required: false
+    choices: ['yes', 'no']
+  from_ip:
+    description:
+      - Source IP address.
+    required: false
+    aliases: ['from', 'src']
+    default: 'any'
+  from_port:
+    description:
+      - Source port.
+    required: false
+  to_ip:
+    description:
+      - Destination IP address.
+    required: false
+    aliases: ['to', 'dest']
+    default: 'any'
+  to_port:
+    description:
+      - Destination port.
+    required: false
+    aliases: ['port']
+  proto:
+    description:
+      - TCP/IP protocol.
+    choices: ['any', 'tcp', 'udp', 'ipv6', 'esp', 'ah']
+    required: false
+  name:
+    description:
+      - Use profile located in C(/etc/ufw/applications.d)
+    required: false
+    aliases: ['app']
+  delete:
+    description:
+      - Delete rule.
+    required: false
+    choices: ['yes', 'no']
+'''
+
+EXAMPLES = '''
+# Allow everything and enable UFW
+ufw: state=enabled policy=allow
+
+# Set logging
+ufw: logging=on
+
+# Sometimes it is desirable to let the sender know when traffic is
+# being denied, rather than simply ignoring it. In these cases, use
+# reject instead of deny. In addition, log rejected connections:
+ufw: rule=reject port=auth log=yes
+
+# ufw supports connection rate limiting, which is useful for protecting
+# against brute-force login attacks. ufw will deny connections if an IP
+# address has attempted to initiate 6 or more connections in the last
+# 30 seconds. See  http://www.debian-administration.org/articles/187
+# for details. Typical usage is:
+ufw: rule=limit port=ssh proto=tcp
+
+# Allow OpenSSH
+ufw: rule=allow name=OpenSSH
+
+# Delete OpenSSH rule
+ufw: rule=allow name=OpenSSH delete=yes
+
+# Deny all access to port 53:
+ufw: rule=deny port=53
+
+# Allow all access to tcp port 80:
+ufw: rule=allow port=80 proto=tcp
+
+# Allow all access from RFC1918 networks to this host:
+ufw: rule=allow src={{ item }}
+with_items:
+- 10.0.0.0/8
+- 172.16.0.0/12
+- 192.168.0.0/16
+
+# Deny access to udp port 514 from host 1.2.3.4:
+ufw: rule=deny proto=udp src=1.2.3.4 port=514
+
+# Allow incoming access to eth0 from 1.2.3.5 port 5469 to 1.2.3.4 port 5469
+ufw: rule=allow interface=eth0 direction=in proto=udp src=1.2.3.5 from_port=5469 dest=1.2.3.4 to_port=5469
+
+# Deny all traffic from the IPv6 2001:db8::/32 to tcp port 25 on this host.
+# Note that IPv6 must be enabled in /etc/default/ufw for IPv6 firewalling to work.
+ufw: rule=deny proto=tcp src=2001:db8::/32 port=25
+'''
+
+from operator import itemgetter
+
+
+def main():
+    module = AnsibleModule(
+        argument_spec = dict(
+            state     = dict(default=None,  choices=['enabled', 'disabled', 'reloaded', 'reset']),
+            default   = dict(default=None,  aliases=['policy'], choices=['allow', 'deny', 'reject']),
+            logging   = dict(default=None,  choices=['on', 'off', 'low', 'medium', 'high', 'full']),
+            direction = dict(default=None,  choices=['in', 'incoming', 'out', 'outgoing']),
+            delete    = dict(default=False, type='bool'),
+            insert    = dict(default=None),
+            rule      = dict(default=None,  choices=['allow', 'deny', 'reject', 'limit']),
+            interface = dict(default=None,  aliases=['if']),
+            log       = dict(default=False, type='bool'),
+            from_ip   = dict(default='any', aliases=['src', 'from']),
+            from_port = dict(default=None),
+            to_ip     = dict(default='any', aliases=['dest', 'to']),
+            to_port   = dict(default=None,  aliases=['port']),
+            proto     = dict(default=None,  aliases=['protocol'], choices=['any', 'tcp', 'udp', 'ipv6', 'esp', 'ah']),
+            app       = dict(default=None,  aliases=['name'])
+        ),
+        supports_check_mode = True,
+        mutually_exclusive = [['app', 'proto', 'logging']]
+    )
+
+    cmds = []
+
+    def execute(cmd):
+        cmd = ' '.join(map(itemgetter(-1), filter(itemgetter(0), cmd)))
+
+        cmds.append(cmd)
+        (rc, out, err) = module.run_command(cmd)
+
+        if rc != 0:
+            module.fail_json(msg=err or out)
+
+    params = module.params
+
+    # Ensure at least one of the command arguments are given
+    command_keys = ['state', 'default', 'rule', 'logging']
+    commands = dict((key, params[key]) for key in command_keys if params[key])
+
+    if len(commands) < 1:
+        module.fail_json(msg="Not any of the command arguments %s given" % commands)
+
+    if('interface' in params and 'direction' not in params):
+      module.fail_json(msg="Direction must be specified when creating a rule on an interface")
+
+    # Ensure ufw is available
+    ufw_bin = module.get_bin_path('ufw', True)
+
+    # Save the pre state and rules in order to recognize changes
+    (_, pre_state, _) = module.run_command(ufw_bin + ' status verbose')
+    (_, pre_rules, _) = module.run_command("grep '^### tuple' /lib/ufw/user*.rules")
+
+    # Execute commands
+    for (command, value) in commands.iteritems():
+        cmd = [[ufw_bin], [module.check_mode, '--dry-run']]
+
+        if command == 'state':
+            states = { 'enabled': 'enable',  'disabled': 'disable',
+                       'reloaded': 'reload', 'reset': 'reset' }
+            execute(cmd + [['-f'], [states[value]]])
+
+        elif command == 'logging':
+            execute(cmd + [[command], [value]])
+
+        elif command == 'default':
+            execute(cmd + [[command], [value], [params['direction']]])
+
+        elif command == 'rule':
+            # Rules are constructed according to the long format
+            #
+            # ufw [--dry-run] [delete] [insert NUM] allow|deny|reject|limit [in|out on INTERFACE] [log|log-all] \
+            #     [from ADDRESS [port PORT]] [to ADDRESS [port PORT]] \
+            #     [proto protocol] [app application]
+            cmd.append([module.boolean(params['delete']), 'delete'])
+            cmd.append([params['insert'], "insert %s" % params['insert']])
+            cmd.append([value])
+            cmd.append([module.boolean(params['log']), 'log'])
+
+            for (key, template) in [('direction', "%s"      ), ('interface', "on %s"   ),
+                                    ('from_ip',   "from %s" ), ('from_port', "port %s" ),
+                                    ('to_ip',     "to %s"   ), ('to_port',   "port %s" ),
+                                    ('proto',     "proto %s"), ('app',       "app '%s'")]:
+
+                value = params[key]
+                cmd.append([value, template % (value)])
+
+            execute(cmd)
+
+    # Get the new state
+    (_, post_state, _) = module.run_command(ufw_bin + ' status verbose')
+    (_, post_rules, _) = module.run_command("grep '^### tuple' /lib/ufw/user*.rules")
+    changed = (pre_state != post_state) or (pre_rules != post_rules)
+
+    return module.exit_json(changed=changed, commands=cmds, msg=post_state.rstrip())
+
+# import module snippets
+from ansible.module_utils.basic import *
+
+main()

=== added symlink 'hooks/client-relation-departed'
=== target is u'hooks.py'
=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2014-07-07 10:22:27 +0000
+++ hooks/hooks.py	2014-07-30 06:36:30 +0000
@@ -4,6 +4,7 @@
 import sys
 import charmhelpers.contrib.ansible
 import charmhelpers.payload.execd
+import charmhelpers.core.host
 
 
 hooks = charmhelpers.contrib.ansible.AnsibleHooks(
@@ -17,8 +18,8 @@
         'start',
         'stop',
         'upgrade-charm',
-        'client-relation-changed',
         'client-relation-joined',
+        'client-relation-departed',
     ])
 
 
@@ -30,6 +31,12 @@
     charmhelpers.contrib.ansible.install_ansible_support(
         from_ppa=False)
 
+    # We copy the backported ansible modules here because they need to be
+    # in place by the time ansible runs any hook.
+    charmhelpers.core.host.rsync(
+        'ansible_module_backports',
+        '/usr/share/ansible')
+
 
 if __name__ == "__main__":
     hooks.execute(sys.argv)

=== modified file 'playbook.yaml'
--- playbook.yaml	2014-07-16 18:38:22 +0000
+++ playbook.yaml	2014-07-30 06:36:30 +0000
@@ -10,9 +10,19 @@
     - name: Restart ElasticSearch
       service: name=elasticsearch state=restarted
 
+  vars:
+    - service_name: "{{ local_unit.split('/')[0] }}"
+    - client_relation_id: "{{ relations['client'].keys()[0] | default('') }}"
+
   tasks:
 
     - include: tasks/install-elasticsearch.yml
+    - include: tasks/setup-ufw.yml
+      tags:
+        - install
+        - upgrade-charm
+        - client-relation-joined
+        - client-relation-departed
     - include: tasks/peer-relations.yml
 
     - name: Update configuration

=== modified file 'tasks/install-elasticsearch.yml'
--- tasks/install-elasticsearch.yml	2014-05-27 14:22:40 +0000
+++ tasks/install-elasticsearch.yml	2014-07-30 06:36:30 +0000
@@ -17,10 +17,13 @@
   when: apt_repository != ""
 
 - name: Install dependent packages.
-  apt: pkg=openjdk-7-jre-headless state=latest update_cache=yes
+  apt: pkg={{ item }} state=latest update_cache=yes
   tags:
     - install
     - upgrade-charm
+  with_items:
+    - openjdk-7-jre-headless
+    - ufw
 
 - name: Check for local elasticsearch.deb in payload.
   stat: path=files/elasticsearch.deb

=== added file 'tasks/setup-ufw.yml'
--- tasks/setup-ufw.yml	1970-01-01 00:00:00 +0000
+++ tasks/setup-ufw.yml	2014-07-30 06:36:30 +0000
@@ -0,0 +1,29 @@
+# XXX 2014-07-08 michael nelson ip6 not supported on image (?)
+# ufw errors unless you switch off ipv6 support. Not sure if it's
+# related to the kernel used on the cloud image, but the actual
+# error is:
+# ip6tables v1.4.21: can't initialize ip6tables table `filter':
+# Table does not exist (do you need to insmod?)
+# Perhaps ip6tables or your kernel needs to be upgraded.
+- name: Update ufw config to avoid error
+  lineinfile: dest=/etc/default/ufw
+              regexp="^IPV6=yes$"
+              line="IPV6=no"
+
+# XXX 2014-07-30 michael nelson: It'd be much nicer if we could
+# just render a config file for ufw, as it would be idempotent.
+# As it is, there isn't a way to do that (afaics), so instead we
+# reset the firewall rules each time based on the current clients.
+- name: Reset firewall
+  ufw: state=reset policy=allow logging=on
+
+- name: Turn on fire wall with logging.
+  ufw: state=enabled policy=allow logging=on
+
+- name: Open the firewall for all clients
+  ufw: rule=allow src={{ item.value['private-address'] }} port=9200 proto=tcp
+  with_dict: relations["client"]["{{ client_relation_id }}"] | default({})
+  when: not item.key.startswith(service_name)
+
+- name: Deny all other requests on 9200
+  ufw: rule=deny port=9200

=== modified file 'unit_tests/test_hooks.py'
--- unit_tests/test_hooks.py	2014-07-07 10:22:27 +0000
+++ unit_tests/test_hooks.py	2014-07-30 06:36:30 +0000
@@ -44,6 +44,14 @@
         execd = self.mock_charmhelpers.payload.execd
         execd.execd_preinstall.assert_called_once_with()
 
+    def test_copys_backported_ansible_modules(self):
+        hooks.execute(['install'])
+
+        rsync = self.mock_charmhelpers.core.host.rsync
+        rsync.assert_called_once_with(
+            'ansible_module_backports',
+            '/usr/share/ansible')
+
 
 class DefaultHooksTestCase(unittest.TestCase):