Merge lp:~veebers/juju-ci-tools/initial-log-forwarding-tests into lp:juju-ci-tools

[Christopher Lee]

Add initial test for log forwarding feature

Adds initial test for the log-forwarding feature.

Handles creation of certificates tied to the rsylog sync as required by the feature

Bootstraps 2 environments, one being the rsyslog sink for logs the other being the emitter of logs.

Has a basic check that exercises the feature, there is further tests to be added but this MP is getting large as it is.

I added a new file 'certificates.py'. I was tempted to create a python package but this isn't a small step and wanted to converse with the team if it was the direction we wanted to go.

I.e. I would propose something like this initially:
  juju_ci_tools/
    __init__.py
    certificates.py

[lp:~veebers/juju-ci-tools/initial-log-forwarding-tests]

1496. By Christopher Lee on 2016-07-14

Fix assertion messages.

1497. By Christopher Lee on 2016-07-14

Remove yet to be used methods.

[Martin Packman (gz)]

Have a bunch of random comments inline. Overall looks like a good approach, my main points of feedback:

* Assess scripts this complex can benefit from a bit more structure. A bunch of cooperating functions gets hard to follow around the 200 line mark. An encapsulating class for each environment seems reasonable.

* A bunch of the setup work is done by manual ssh steps on charms. Ideally, most of this complexity is encapsulated in the charm instead.

* I'd generally have more unit tests, but can see how a bunch of the current code is not amenable to it.

[Martin Packman (gz)]

This time with the comments...

[Christopher Lee (veebers)]

Responded to comments ans made some changes. Still have a couple of changes to make.

[lp:~veebers/juju-ci-tools/initial-log-forwarding-tests]

1498. By Christopher Lee on 2016-07-19

Grammar fix and import fix.

1499. By Christopher Lee on 2016-07-19

Better naming for boostrap managers.

1500. By Christopher Lee on 2016-07-19

Change certificate details to not be a getter function.

[Martin Packman (gz)]

Replied to some comments.

[lp:~veebers/juju-ci-tools/initial-log-forwarding-tests]

1501. By Christopher Lee on 2016-07-25

Merge trunk

1502. By Christopher Lee on 2016-07-26

Rename and reduce assess function

1503. By Christopher Lee on 2016-07-28

Using python script run on remote machine for logs check.

1504. By Christopher Lee on 2016-07-28

Add failling test for get_controller_uuid

1505. By Christopher Lee on 2016-07-28

Make test pass for added get_controller_uuid

1506. By Christopher Lee on 2016-07-28

Modify test to use get_controller_uuid

1507. By Christopher Lee on 2016-07-28

Cater for ipv6 addresses + port. Incl. tests.

1508. By Christopher Lee on 2016-07-28

Fix remote python script.

[Martin Packman (gz)]

Couple of notes from eyeballing this again, see inline comments.

[Christopher Lee (veebers)]

Made the suggested changes.

[lp:~veebers/juju-ci-tools/initial-log-forwarding-tests]

1509. By Christopher Lee on 2016-08-03

Put check logic into separate file with tests.

[Martin Packman (gz)]

Some comments on how to write the new code a little better.

[lp:~veebers/juju-ci-tools/initial-log-forwarding-tests]

1510. By Christopher Lee on 2016-08-04

Improved file existence checks and handling.

[Martin Packman (gz)]

Thanks! Lets land it.

review: Approve

Unmerged revisions

1510. By Christopher Lee on 2016-08-04

Improved file existence checks and handling.

=== added file 'assess_log_forward.py'
--- assess_log_forward.py	1970-01-01 00:00:00 +0000
+++ assess_log_forward.py	2016-08-04 22:17:14 +0000
@@ -0,0 +1,282 @@
+#!/usr/bin/env python
+"""Test Juju's log forwarding feature.
+
+Log forwarding allows a controller to forward syslog from all models of a
+controller to a syslog host via TCP (using SSL).
+
+"""
+
+from __future__ import print_function
+
+import argparse
+import logging
+import os
+import re
+import sys
+import socket
+import subprocess
+from textwrap import dedent
+
+from assess_model_migration import get_bootstrap_managers
+import certificates
+from utility import (
+    add_basic_testing_arguments,
+    configure_logging,
+    JujuAssertionError,
+    temp_dir,
+)
+
+
+__metaclass__ = type
+
+
+log = logging.getLogger("assess_log_forward")
+
+
+def assess_log_forward(bs_dummy, bs_rsyslog, upload_tools):
+    """Ensure logs are forwarded after forwarding enabled after bootstrapping.
+
+    Given 2 controllers set rsyslog and dummy:
+      - setup rsyslog with secure details
+      - Enable log forwarding on dummy
+      - Ensure intial logs are present in the rsyslog sinks logs
+
+    """
+    with bs_rsyslog.booted_context(upload_tools):
+        log.info('Bootstrapped rsyslog environment')
+        rsyslog = bs_rsyslog.client
+        rsyslog_details = deploy_rsyslog(rsyslog)
+
+        update_client_config(bs_dummy.client, rsyslog_details)
+
+        with bs_dummy.existing_booted_context(upload_tools):
+            log.info('Bootstrapped dummy environment')
+            dummy_client = bs_dummy.client
+
+            # ensure turning on log-fwd emits logs to the client.
+            # Should I ensure that nothing turns up beforehand
+            ensure_enabling_log_forwarding_forwards_previous_messages(
+                rsyslog, dummy_client)
+
+
+def ensure_enabling_log_forwarding_forwards_previous_messages(rsyslog, dummy):
+    """Assert when enabled log forwarding forwards messages from log start."""
+    uuid = dummy.get_controller_uuid()
+
+    enable_log_forwarding(dummy)
+    assert_initial_message_forwarded(rsyslog, uuid)
+
+
+def assert_initial_message_forwarded(rsyslog, uuid):
+    """Assert that mention of the sources logs appear in the sinks logging.
+
+    Given a rsyslog sink and an output source assert that logging details from
+    the source appear in the sinks logging.
+    Attempt a check over a period of time (10 seconds).
+
+    :returns: As soon as the expected message appears.
+    :raises JujuAssertionError: If the expected message does not appear in the
+      given timeframe.
+    :raises JujuAssertionError: If the log message check fails in an unexpected
+      way.
+
+    """
+    check_string = get_assert_regex(uuid)
+    unit_machine = 'rsyslog/0'
+
+    remote_script_path = create_check_script_on_unit(rsyslog, unit_machine)
+
+    try:
+        rsyslog.juju(
+            'ssh',
+            (
+                unit_machine,
+                'sudo',
+                'python',
+                remote_script_path,
+                check_string,
+                '/var/log/syslog'))
+        log.info('Check script passed on target machine.')
+    except subprocess.CalledProcessError:
+        # This is where a failure happened
+        raise JujuAssertionError('Forwarded log message never appeared.')
+
+
+def create_check_script_on_unit(client, unit_machine):
+    script_path = os.path.join(os.path.dirname(__file__), 'log_check.py')
+    script_dest_path = os.path.join('/tmp', os.path.basename(script_path))
+    client.juju(
+        'scp',
+        (script_path, '{}:{}'.format(unit_machine, script_dest_path)))
+    return script_dest_path
+
+
+def get_assert_regex(raw_uuid, message=None):
+    """Create a regex string to check syslog file.
+
+    If message is supplied it is expected to be escaped as needed (i.e. spaces)
+    no further massaging will be done to the message string.
+
+    """
+    # Maybe over simplified removing the last 8 characters
+    uuid = re.escape(raw_uuid)
+    short_uuid = re.escape(raw_uuid[:-8])
+    date_check = '[A-Z][a-z]{,2}\ +[0-9]+\ +[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}'
+    machine = 'machine-0.{}'.format(uuid)
+    agent = 'jujud-machine-agent-{}'.format(short_uuid)
+    message = message or '.*'
+
+    return '"^{datecheck}\ {machine}\ {agent}\ {message}$"'.format(
+        datecheck=date_check,
+        machine=machine,
+        agent=agent,
+        message=message)
+
+
+def enable_log_forwarding(client):
+    client.juju(
+        'set-model-config',
+        ('-m', 'controller', 'logforward-enabled=true'), include_e=False)
+
+
+def update_client_config(client, rsyslog_details):
+    client.env.config['logforward-enabled'] = False
+    client.env.config.update(rsyslog_details)
+
+
+def deploy_rsyslog(client):
+    """Deploy and setup the rsyslog charm on client.
+
+    :returns: Configuration details needed: cert, ca, key and ip:port.
+
+    """
+    app_name = 'rsyslog'
+    client.deploy('rsyslog', (app_name))
+    client.wait_for_started()
+    client.juju('set-config', (app_name, 'protocol="tcp"'))
+    client.juju('expose', app_name)
+
+    return setup_tls_rsyslog(client, app_name)
+
+
+def setup_tls_rsyslog(client, app_name):
+    unit_machine = '{}/0'.format(app_name)
+
+    ip_address = get_unit_ipaddress(client, unit_machine)
+
+    client.juju(
+        'ssh',
+        (unit_machine, 'sudo apt-get install rsyslog-gnutls'))
+
+    with temp_dir() as config_dir:
+        install_rsyslog_config(client, config_dir, unit_machine)
+        rsyslog_details = install_certificates(
+            client, config_dir, ip_address, unit_machine)
+
+    # restart rsyslog to take into affect all changes
+    client.juju('ssh', (unit_machine, 'sudo', 'service', 'rsyslog', 'restart'))
+
+    return rsyslog_details
+
+
+def install_certificates(client, config_dir, ip_address, unit_machine):
+    cert, key = certificates.create_certificate(config_dir, ip_address)
+
+    # Write contents to file to scp across
+    ca_file = os.path.join(config_dir, 'ca.pem')
+    with open(ca_file, 'wt') as f:
+        f.write(certificates.ca_pem_contents)
+
+    scp_command = (
+        '--', cert, key, ca_file, '{unit}:/home/ubuntu/'.format(
+            unit=unit_machine))
+    client.juju('scp', scp_command)
+
+    return _get_rsyslog_details(cert, key, ip_address)
+
+
+def _get_rsyslog_details(cert_file, key_file, ip_address):
+    with open(cert_file, 'rt') as f:
+        cert_contents = f.read()
+    with open(key_file, 'rt') as f:
+        key_contents = f.read()
+
+    return {
+        'syslog-host': '{}'.format(add_port_to_ip(ip_address, '10514')),
+        'syslog-ca-cert': certificates.ca_pem_contents,
+        'syslog-client-cert': cert_contents,
+        'syslog-client-key': key_contents
+    }
+
+
+def add_port_to_ip(ip_address, port):
+    """Return an ipv4/ipv6 address with port added to `ip_address`."""
+    try:
+        socket.inet_aton(ip_address)
+        return '{}:{}'.format(ip_address, port)
+    except socket.error:
+        try:
+            socket.inet_pton(socket.AF_INET6, ip_address)
+            return '[{}]:{}'.format(ip_address, port)
+        except socket.error:
+            pass
+    raise ValueError(
+        'IP Address "{}" is neither an ipv4 or ipv6 address.'.format(
+            ip_address))
+
+
+def install_rsyslog_config(client, config_dir, unit_machine):
+    config = write_rsyslog_config_file(config_dir)
+    client.juju('scp', (config, '{unit}:/tmp'.format(unit=unit_machine)))
+    client.juju(
+        'ssh',
+        (unit_machine, 'sudo', 'mv', '/tmp/{}'.format(
+            os.path.basename(config)), '/etc/rsyslog.d/'))
+
+
+def get_unit_ipaddress(client, unit_name):
+    status = client.get_status()
+    return status.get_unit(unit_name)['public-address']
+
+
+def write_rsyslog_config_file(tmp_dir):
+    """Write rsyslog config file to `tmp_dir`/10-securelogging.conf."""
+    config = dedent("""\
+    # make gtls driver the default
+    $DefaultNetstreamDriver gtls
+
+    # certificate files
+    $DefaultNetstreamDriverCAFile /home/ubuntu/ca.pem
+    $DefaultNetstreamDriverCertFile /home/ubuntu/cert.pem
+    $DefaultNetstreamDriverKeyFile /home/ubuntu/key.pem
+
+    $ModLoad imtcp # load TCP listener
+    $InputTCPServerStreamDriverAuthMode x509/name
+    $InputTCPServerStreamDriverPermittedPeer anyServer
+    $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
+    $InputTCPServerRun 10514 # port 10514
+    """)
+    config_path = os.path.join(tmp_dir, '10-securelogging.conf')
+    with open(config_path, 'wt') as f:
+        f.write(config)
+    return config_path
+
+
+def parse_args(argv):
+    """Parse all arguments."""
+    parser = argparse.ArgumentParser(
+        description="Test log forwarding of logs.")
+    add_basic_testing_arguments(parser)
+    return parser.parse_args(argv)
+
+
+def main(argv=None):
+    args = parse_args(argv)
+    configure_logging(args.verbose)
+    bs_dummy, bs_rsyslog = get_bootstrap_managers(args)
+    assess_log_forward(bs_dummy, bs_rsyslog, args.upload_tools)
+    return 0
+
+
+if __name__ == '__main__':
+    sys.exit(main())
 
=== added file 'certificates.py'
--- certificates.py	1970-01-01 00:00:00 +0000
+++ certificates.py	2016-08-04 22:17:14 +0000
@@ -0,0 +1,139 @@
+from OpenSSL import crypto
+import os
+from textwrap import dedent
+
+
+def create_certificate(target_dir, ip_address):
+    """Generate a cert and key file incl. IP SAN for `ip_address`
+
+    Creates a cert.pem and key.pem file signed with a known ca cert.
+    The generated cert will contain a IP SAN (subject alternative name) that
+    includes the ip address of the server. This is required for log-forwarding.
+
+    :return: tuple containing generated cert, key filepath pair
+
+    """
+    ip_address = 'IP:{}'.format(ip_address)
+
+    key = crypto.PKey()
+    key.generate_key(crypto.TYPE_RSA, 2048)
+
+    csr_contents = generate_csr(target_dir, key, ip_address)
+    req = crypto.load_certificate_request(crypto.FILETYPE_PEM, csr_contents)
+
+    ca_cert = crypto.load_certificate(
+        crypto.FILETYPE_PEM, ca_pem_contents)
+    ca_key = crypto.load_privatekey(
+        crypto.FILETYPE_PEM, ca_key_pem_contents)
+
+    cert = crypto.X509()
+    cert.set_version(0x2)
+    cert.set_subject(req.get_subject())
+    cert.set_serial_number(1)
+    cert.gmtime_adj_notBefore(0)
+    cert.gmtime_adj_notAfter(24 * 60 * 60)
+    cert.set_issuer(ca_cert.get_subject())
+    cert.set_pubkey(req.get_pubkey())
+    cert.add_extensions([
+        crypto.X509Extension('subjectAltName', False, ip_address),
+        crypto.X509Extension(
+            'extendedKeyUsage', False, 'clientAuth, serverAuth'),
+        crypto.X509Extension(
+            'keyUsage', True, 'keyEncipherment'),
+    ])
+    cert.sign(ca_key, "sha256")
+
+    cert_filepath = os.path.join(target_dir, 'cert.pem')
+    with open(cert_filepath, 'wt') as f:
+        f.write(crypto.dump_certificate(crypto.FILETYPE_PEM, cert))
+
+    key_filepath = os.path.join(target_dir, 'key.pem')
+    with open(key_filepath, 'wt') as f:
+        f.write(crypto.dump_privatekey(crypto.FILETYPE_PEM, key))
+
+    return (cert_filepath, key_filepath)
+
+
+def generate_csr(target_dir, key, ip_address):
+    req = crypto.X509Req()
+    req.set_version(0x2)
+    req.get_subject().CN = "anyServer"
+    # Add the IP SAN
+    req.add_extensions([
+        crypto.X509Extension("subjectAltName", False, ip_address)
+    ])
+    req.set_pubkey(key)
+    req.sign(key, "sha256")
+
+    return crypto.dump_certificate_request(crypto.FILETYPE_PEM, req)
+
+
+ca_pem_contents = dedent("""\
+    -----BEGIN CERTIFICATE-----
+    MIIEFTCCAn2gAwIBAgIBBzANBgkqhkiG9w0BAQsFADAjMRIwEAYDVQQDEwlhbnlT
+    ZXJ2ZXIxDTALBgNVBAoTBGp1anUwHhcNMTYwNzExMDQyOTM1WhcNMjYwNzA5MDQy
+    OTM1WjAjMRIwEAYDVQQDEwlhbnlTZXJ2ZXIxDTALBgNVBAoTBGp1anUwggGiMA0G
+    CSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQCn6OxY33yAirABoE4UaZJBOnQORIzC
+    125R71E2TG5gSHjHKA70L0C3dgyWhW9wcyhUbXBuz8Oep2J7kHvzuUPw2AWXI+Y2
+    c0afWVqfj5kuyUpGhXsqylyf7NDPFs8hwGA6ZCFS3oUAvX8awsVucklxGeZNXZNK
+    ZFilXKaX1Z3soORmKFZzVfDRqDuofZ2E0tmPh9C5gQ8qswjdBnTrj+0rCnvNekO0
+    aND6AlkBHU+87pvcax0uUF6PYkXxPikKk1ftCQSII5oB5ksAtRpcZsYl5hT3U/t1
+    DOA7c35RuIx7ogkcXP9jZ6J2tkmX+GMtUF29KEEnVCht32VDX+C3yS6lbfQB4oDt
+    Yp3wXRY/LXTW7XTUrhoXB4nkYbw59gis5Cr7zDtUpiWFVYgy/kbxalljSM4N3w2i
+    dtfxJHYjTfK98124qbCBb4A4ZNBJE2jy//lSIcIMXJv1LXQtTqR4rO1j6TBurohF
+    NmUYpy3Zv7gn2CkfX6QfNFIj8elKT6dd+RUCAwEAAaNUMFIwDwYDVR0TAQH/BAUw
+    AwEB/zAPBgNVHREECDAGhwQKwoylMA8GA1UdDwEB/wQFAwMHBAAwHQYDVR0OBBYE
+    FP+v8GAqHiUCIygXbwWzbUhl/22DMA0GCSqGSIb3DQEBCwUAA4IBgQBVYKeT1O2M
+    U3OPOy0IwqcA1/64rS1GlRmiw+papmjsy3aru03r8igahnbFd7wQawHaCScXbI/n
+    OAPT4PDGXn6b71t4uHwWaM8wde159RO3G32N/VfhV6LPRUQunmAZh5QcJK6wWpYu
+    B1f0dPkU+Q1AfX12oTOX/ld2/o7jaVswHoHoW6K2WQmwzlRQ953J+RJ7jXfrYDKl
+    OAp3Hb69wAN4Ayc1s92iYUwV5q8UaHQoskHOLWJu964yFBHL8SLe6TLD+Jjv05Mc
+    Ca7NKq/n25VTDNNaXl5MCNZ048m/GGHfktxxCddaF2grhC5HTUetwkq026PE0Wcq
+    P+cDrIq6uTA25QqyBYistSa/7z2o0NBi56ySRqxlP2J2TPFZyOb+ZiA4EgYY5no5
+    u2E+WuKZLVWl7eaQYOHgfYzFf3CvalSBwIjNynRwD/2Ebk7K29GPrIugb3V2+Vwh
+    rltUXOHUkFGjEHIhr8zixfCxh5OzPJMnJwCZZRYzMO0/0Gw7ll9DmH0=
+    -----END CERTIFICATE-----
+    """)
+
+
+ca_key_pem_contents = dedent("""\
+    -----BEGIN RSA PRIVATE KEY-----
+    MIIG4wIBAAKCAYEAp+jsWN98gIqwAaBOFGmSQTp0DkSMwtduUe9RNkxuYEh4xygO
+    9C9At3YMloVvcHMoVG1wbs/Dnqdie5B787lD8NgFlyPmNnNGn1lan4+ZLslKRoV7
+    Kspcn+zQzxbPIcBgOmQhUt6FAL1/GsLFbnJJcRnmTV2TSmRYpVyml9Wd7KDkZihW
+    c1Xw0ag7qH2dhNLZj4fQuYEPKrMI3QZ064/tKwp7zXpDtGjQ+gJZAR1PvO6b3Gsd
+    LlBej2JF8T4pCpNX7QkEiCOaAeZLALUaXGbGJeYU91P7dQzgO3N+UbiMe6IJHFz/
+    Y2eidrZJl/hjLVBdvShBJ1Qobd9lQ1/gt8kupW30AeKA7WKd8F0WPy101u101K4a
+    FweJ5GG8OfYIrOQq+8w7VKYlhVWIMv5G8WpZY0jODd8NonbX8SR2I03yvfNduKmw
+    gW+AOGTQSRNo8v/5UiHCDFyb9S10LU6keKztY+kwbq6IRTZlGKct2b+4J9gpH1+k
+    HzRSI/HpSk+nXfkVAgMBAAECggGAJigjVYrr8xYRKz1voOngx5vt9bQUPM7SDiKR
+    VQKHbq/panCq/Uijr01PTQFjsq0otA7upu/l527oTWYnFNq8GsYsdw08apFFsj6O
+    /oWWbPBnRaFdvPqhk+IwDW+EgIoEFCDfBcL1fJaThNRQI2orUF1vXZNvPk+RaXql
+    jQmJStXBMYnnI2ybPjm53O821ZFIyXo2r4Epni1zTS8DcOiTH93RBn/LVPsgyj+w
+    VDWCAlBC8RMSXYz8AB93/3t9vh5/VTE8qRC9j6lqTxNsUYlCsHuB/j6A7XqFU6U7
+    BVkKUHXRKo2nNcKwjsfPlnk/M41JT/N5RIpTbXRiBgZklIcXxxWdYDGD6M7n2YiP
+    dMwmLZIxPRVp7LTQIxrztkqL5Kp/X9DasI6BPCgifxm4spvjMn5X+k5x4E6GABC2
+    lx/cgriOl+nxgsy4372Kpt62srPRu4Vajr6DDH6nR1O0vxqu4ifawoe7YAUzXzvi
+    5kFWNzpnQ9pZ9s8iW0xP4eAuVZydAoHBAMEToj0he4vY5NyH/axf6u9BA/gkXn4D
+    z38uYykYLr5b8BdEpbB0xZ/LgFOq45ZJcEYo0NjPLgiKuvtvZAKXm0Pka4a8D9Cp
+    NhhoIN9iarZxgDkwvPX2VO1oGB8G/C5WlB2Y0P7QW9wxXZjA0KOkSJEdLP9kBvuQ
+    s/eezIYUiM6upvqPqwKtniMYH1Dz3pApId/APUre0Qo52ITJGr6D849BfMqKYb5Z
+    4ifBUeztydZy8goNHIv4yERUVGoHVviWpwKBwQDeoZ+EGqv010U7ioMIhkJnt4CY
+    CrAHOFJye+Th1wRHGGFy/UOe8SwxwZPAbexH/+HgC5IQ9FSx5SIDuaSWmjOd0DUi
+    Lih2+J3T29haP2259gCvy9UtU+MGW6hP+bhdyJl1SmxSetfDAToAA5tBTSjcu4ea
+    8bKZwm7gHwxnXMuuGkkIUNSul1P9FwUEi3ZaefF3LN3P03e0T93n97DWCKA5yL2w
+    tx7Y8o8AGyBaajPj9S8jLvw8bMzaSuXizucL5eMCgcBdX29gfObQtO3JMQMe76wg
+    VKLkyEHiU1lvujE+WHGSoce0mQBAG9jO9I106PnzXkSryWVm1JsAiobuvenxzvvJ
+    k5fkquJDGPIOT51GKsRMwwstnUJk+OINhf/UUX53smsi/RplgMJL9Ju9GdJMsVBe
+    zWtLf0ZZNpuyLtveI+QdgB1Eo2Iig3AsrKfIcIe71AiLut5pbORPO7ZYUSFb7VhG
+    eXcuREoM0k8qxrUmDcFEsoYXEkwx7Ph9AwNn23DV+5UCgcEA2ojWN2ui/czOJdsa
+    MqTvzDWBoj1je0LbE4vwKYvRpCQXjDN1TDC6zACTk2GTfT19MFrLP59G//TGhdeV
+    60tkfXXiojGjAN2ct1jnL/dxMwh6thWkpUDh6dzRA+hCBLUjhdHPMMtqvf2XPGpN
+    3TTrdnkSbJLyWSJVieSQXWnmeXlN1T7a9qKPDDGreEGZpMhssSo2dYnDyBhZ4Bjv
+    2blP5kjZgvzN5/F5U4ZNJNN5KjwD0EqPyJSYJXM943xrqe83AoHAUYcDXY+TEpvQ
+    WSHib0P+0yX4uZblgAqWk6nPKFIS1mw4mCO71vRHbxztA9gmqxhdSU2aDhHBslIg
+    50eGW9aaTaR6M6vsULA4danJso8Fzgiaz3oxOwSkxBdIu1F0Mr6JlI5PEN21vKXX
+    tsiC2JJEasQbEbNLA5X4hX/jXWwPw0JGMW6UR6RaMHevA09579COUFrtEguZfDi6
+    1xP72bo/RzQ1cWLjb5QVkf155q/BpzrWYQJwo/8TEIL33XZcMES5
+    -----END RSA PRIVATE KEY-----
+    """)
 
=== modified file 'jujupy.py'
--- jujupy.py	2016-08-03 21:23:26 +0000
+++ jujupy.py	2016-08-04 22:17:14 +0000
@@ -1476,6 +1476,13 @@
             env = self.env.clone(model_name=name)
             return self.clone(env=env)
 
+    def get_controller_uuid(self):
+        name = self.env.controller.name
+        output_yaml = self.get_juju_output(
+            'show-controller', '--format', 'yaml', include_e=False)
+        output = yaml.safe_load(output_yaml)
+        return output[name]['details']['uuid']
+
     def get_controller_client(self):
         """Return a client for the controller model.  May return self.
 
 
=== added file 'log_check.py'
--- log_check.py	1970-01-01 00:00:00 +0000
+++ log_check.py	2016-08-04 22:17:14 +0000
@@ -0,0 +1,55 @@
+#!/usr/bin/env python
+"""Simple looped check over provided file for regex content."""
+from __future__ import print_function
+
+import argparse
+import os
+import subprocess
+import sys
+import time
+
+
+class check_result:
+    success = 0
+    failure = 1
+    exception = 2
+
+
+def check_file(check_string, file_path):
+    print('Checking for:\n{}'.format(check_string))
+    for _ in range(0, 10):
+        try:
+            with open(file_path, 'r') as f:
+                subprocess.check_call(['sudo', 'egrep', check_string], stdin=f)
+            print('Log content found. No need to continue.')
+            return check_result.success
+        except subprocess.CalledProcessError as e:
+            if e.returncode == 1:
+                time.sleep(1)
+            else:
+                return check_result.exception
+    print('Unexpected error with file check.')
+    return check_result.failure
+
+
+def parse_args(argv=None):
+    parser = argparse.ArgumentParser(
+        description='File content check.')
+    parser.add_argument(
+        'regex', help='Regex string to check file with.')
+    parser.add_argument(
+        'file_path', help='Path to file to check.')
+    return parser.parse_args(argv)
+
+
+def main(argv=None):
+    args = parse_args(argv)
+    try:
+        return check_file(args.regex, args.file_path)
+    except EnvironmentError as e:
+        print('Cannot open file "{}": {}'.format(args.file_path, str(e)))
+        return check_result.exception
+
+
+if __name__ == '__main__':
+    sys.exit(main())
 
=== added file 'tests/test_assess_log_forward.py'
--- tests/test_assess_log_forward.py	1970-01-01 00:00:00 +0000
+++ tests/test_assess_log_forward.py	2016-08-04 22:17:14 +0000
@@ -0,0 +1,96 @@
+"""Tests for assess_log_forward module."""
+
+import argparse
+from mock import patch
+import re
+import StringIO
+
+import assess_log_forward as alf
+from tests import (
+    parse_error,
+    TestCase,
+)
+
+
+class TestParseArgs(TestCase):
+
+    def test_common_args(self):
+        args = alf.parse_args(
+            ['an-env', '/bin/juju', '/tmp/logs', 'an-env-mod'])
+        self.assertEqual(
+            args,
+            argparse.Namespace(
+                env='an-env',
+                juju_bin='/bin/juju',
+                temp_env_name='an-env-mod',
+                debug=False,
+                agent_stream=None,
+                agent_url=None,
+                bootstrap_host=None,
+                keep_env=False,
+                logs='/tmp/logs',
+                machine=[],
+                region=None,
+                series=None,
+                upload_tools=False,
+                verbose=20))
+
+    def test_help(self):
+        fake_stdout = StringIO.StringIO()
+        with parse_error(self) as fake_stderr:
+            with patch("sys.stdout", fake_stdout):
+                alf.parse_args(["--help"])
+        self.assertEqual("", fake_stderr.getvalue())
+        self.assertIn(
+            'Test log forwarding of logs.',
+            fake_stdout.getvalue())
+
+
+class TestAssertRegex(TestCase):
+
+    def test_default_message_check(self):
+        self.assertTrue(
+            alf.get_assert_regex('').endswith('.*$"'))
+
+    def test_fails_when_uuid_doesnt_match(self):
+        uuid = 'fail'
+        check = alf.get_assert_regex(uuid)
+        failing_string = 'abc'
+
+        self.assertIsNone(re.search(check, failing_string))
+
+    def test_succeeds_with_matching_uuid(self):
+        uuid = '1234567812345678'
+        short_uuid = uuid[:-8]
+        check = alf.get_assert_regex(
+            uuid, message='abc').rstrip('"').strip('"')
+        success = 'Jul 13 00:00:00 machine-0.{} '\
+                  'jujud-machine-agent-{} abc'.format(
+                      uuid, short_uuid)
+
+        self.assertIsNotNone(
+            re.search(check, success))
+
+
+class TestAddPortToIP(TestCase):
+    def test_adds_port_to_ipv4(self):
+        ip_address = '192.168.1.1'
+        port = '123'
+        expected = '192.168.1.1:123'
+        self.assertEqual(
+            alf.add_port_to_ip(ip_address, port),
+            expected
+        )
+
+    def test_adds_port_to_ipv6(self):
+        ip_address = '1fff:0:a88:85a3::ac1f'
+        port = '123'
+        expected = '[1fff:0:a88:85a3::ac1f]:123'
+        self.assertEqual(
+            alf.add_port_to_ip(ip_address, port),
+            expected
+        )
+
+    def test_raises_ValueError_on_invalid_address(self):
+        with self.assertRaises(ValueError):
+            alf.add_port_to_ip('abc', 'abc')
 
=== modified file 'tests/test_jujupy.py'
--- tests/test_jujupy.py	2016-08-03 21:23:26 +0000
+++ tests/test_jujupy.py	2016-08-04 22:17:14 +0000
@@ -2343,6 +2343,35 @@
             controller_name = client.get_controller_model_name()
         self.assertEqual('controller', controller_name)
 
+    def test_get_controller_uuid_returns_uuid(self):
+        controller_uuid = 'eb67e1eb-6c54-45f5-8b6a-b6243be97202'
+        yaml_string = dedent("""\
+        foo:
+          details:
+            uuid: {uuid}
+            api-endpoints: ['10.194.140.213:17070']
+            cloud: lxd
+            region: localhost
+          models:
+            controller:
+              uuid: {uuid}
+            default:
+              uuid: 772cdd39-b454-4bd5-8704-dc9aa9ff1750
+          current-model: default
+          account:
+            user: admin@local
+          bootstrap-config:
+            config:
+            cloud: lxd
+            cloud-type: lxd
+            region: localhost""".format(uuid=controller_uuid))
+        client = EnvJujuClient(JujuData('foo'), None, None)
+        with patch.object(client, 'get_juju_output', return_value=yaml_string):
+            self.assertEqual(
+                client.get_controller_uuid(),
+                controller_uuid
+            )
+
     def test_get_controller_client(self):
         client = EnvJujuClient(
             JujuData('foo', {'bar': 'baz'}, 'myhome'), None, None)
 
=== added file 'tests/test_log_check.py'
--- tests/test_log_check.py	1970-01-01 00:00:00 +0000
+++ tests/test_log_check.py	2016-08-04 22:17:14 +0000
@@ -0,0 +1,53 @@
+"""Tests for log_check script."""
+
+from mock import patch
+import subprocess
+
+import log_check as lc
+from tests import TestCase
+
+
+class TestCheckFile(TestCase):
+
+    regex = 'test123'
+    file_path = '/tmp/file.log'
+
+    def test_calls_check_call(self):
+        with patch.object(lc.subprocess, 'check_call') as m_checkcall:
+            lc.check_file(self.regex, self.file_path)
+
+            m_checkcall.assert_called_once_with(
+                ['sudo', 'egrep', self.regex, self.file_path])
+
+    def test_fails_after_attempting_multiple_times(self):
+        with patch.object(lc.subprocess, 'check_call') as m_checkcall:
+            m_checkcall.side_effect = subprocess.CalledProcessError(
+                1, ['sudo', 'egrep', self.regex, self.file_path])
+            with patch.object(lc.time, 'sleep') as m_sleep:
+                self.assertEqual(
+                    lc.check_file(self.regex, self.file_path),
+                    lc.check_result.failure)
+                self.assertEqual(m_sleep.call_count, 10)
+
+    def test_fails_when_meeting_unexpected_outcome(self):
+        with patch.object(lc.subprocess, 'check_call') as m_checkcall:
+            m_checkcall.side_effect = subprocess.CalledProcessError(
+                -1, ['sudo', 'egrep', self.regex, self.file_path])
+            self.assertEqual(
+                lc.check_file(self.regex, self.file_path),
+                lc.check_result.exception)
+
+    def test_succeeds_when_regex_found(self):
+        with patch.object(lc.subprocess, 'check_call'):
+            self.assertEqual(
+                lc.check_file(self.regex, self.file_path),
+                lc.check_result.success)
+
+
+class TestParseArgs(TestCase):
+
+    def test_basic_args(self):
+        args = ['test .*', '/tmp/log.file']
+        parsed = lc.parse_args(args)
+        self.assertEqual(parsed.regex, 'test .*')
+        self.assertEqual(parsed.file_path, '/tmp/log.file')