Merge lp:~dpb/charms/trusty/landscape-client/fix-integration-tests into lp:~landscape/charms/trusty/landscape-client/trunk

[David Britton]

Get our integration tests back up to snuff. You can test like:

virtualenv bundletester
source bundletester/bin/activate
pip install bundletester
juju env <where you want to be>
bundletester

FYI -- there is nothing fancy about bundletester, it's just a runner and what ~charmers is moving toward for standardized testing of charms/bundles. You can achieve the same functionality by doing this:

make test
make lint
charm proof
./test/00-*
./test/10-*

FYI 2 -- you can see the results of testing here:

https://ci.lscape.net/view/Landscape%20Client/job/landscape-client-charm-test/

[lp:~dpb/charms/trusty/landscape-client/fix-integration-tests]

53. By David Britton on 2014-12-08

add python-amulet to dependency list

54. By David Britton on 2014-12-08

change logic in 00-setup, add amulet to test.yaml (which I don't think is used, but the instructions say to update)

55. By David Britton on 2014-12-08

allow overriding origin, it was so small, I'm including it.

[Andreas Hasenack (ahasenack)]

A few comments inline. I'm not very fond of using staging as a test bed for this, but right now I don't have a better idea other than deploying lds for real, which is out of scope for this.

review: Approve

[Andreas Hasenack (ahasenack)]

I wonder, however, if #is won't complain about us temporarily having root on whatever machine or container that is running this. This could be a big nono.

[Andreas Hasenack (ahasenack)]

If we proceed with staging as is, I'd suggest to create another admin there, with very strict permissions (i.e., only to reject computers), and use those credentials in the jenkins cleanup job.

[David Britton (dpb)]

> If we proceed with staging as is, I'd suggest to create another admin there,
> with very strict permissions (i.e., only to reject computers), and use those
> credentials in the jenkins cleanup job.

This part is done, thanks. Agreed that this is not the best approach (using staging), but other than deploying LDS alongside the client for the integration test (which would make it a much longer and more complicated process), I can't see another good option for actually testing it can register.

As we saw with the recent breakage, registering is the only way to test some of the recent juju-info integration stuff.

[lp:~dpb/charms/trusty/landscape-client/fix-integration-tests]

56. By David Britton on 2014-12-12

andreas[diff]: adding flake8, landscape-common, increasing timeouts

57. By David Britton on 2014-12-12

andreas[diff]: Added comment about how amulet works with charm:landscape-client

[Chad Smith (chad.smith)]

+1 on this branch with 2 minor comments.
Also we shouldn't have related the branch to lp:1399314 right? That bug is a landscape-server charm issue.

review: Approve

[lp:~dpb/charms/trusty/landscape-client/fix-integration-tests]

58. By David Britton on 2014-12-17

remove mention of bundltester from TESTING.

59. By David Britton on 2014-12-17

blackboxsw[diff]: explination for env var added to 'TESTING'

60. By David Britton on 2014-12-17

blackboxsw[diff]: remove useless tail, add s to succes

=== modified file '.bzrignore'
--- .bzrignore	2014-07-16 08:42:43 +0000
+++ .bzrignore	2014-12-17 23:25:40 +0000
@@ -1,1 +1,3 @@
 bin/
+.venv
+hooks/_trial_temp

=== modified file 'TESTING'
--- TESTING	2013-04-29 20:40:39 +0000
+++ TESTING	2014-12-17 23:25:40 +0000
@@ -1,5 +1,17 @@
-Simple end-to-end testing can be performed by:
-
-  cd tests
-  make build
-  ./test.sh
+Simple end-to-end testing can be performed by running the files in order:
+
+  ./tests/00-*
+  ./tests/10-*
+  ...
+
+Environment overrides for end-to-end testing:
+
+  LANDSCAPE_CLIENT_CHARM_ORIGIN - will override the 'origin' juju config
+
+Also other tests are available and should pass:
+
+  make lint
+  make test
+  charm proof
+
+Enjoy!

=== modified file 'config.yaml'
--- config.yaml	2014-10-08 09:28:05 +0000
+++ config.yaml	2014-12-17 23:25:40 +0000
@@ -14,25 +14,30 @@
       The directory to store data files in (default:
       '/var/lib/landscape/client/').
     type: string
+    default:
   log-dir:
     description: |
       The directory to write log files to (default:
       '/var/log/landscape').
     type: string
+    default:
   log-level:
     description: |
       One of debug, info, warning, error or critical.
     type: string
+    default:
   url:
     description: |
       The message server URL to connect to. Normally
       https://fqdn/message-system.
     type: string
+    default:
   ping-url:
     description: |
       The ping server URL to perform lightweight exchange initiation with.
       Normally http://fqdn/ping.
     type: string
+    default:
   ssl-public-key:
     description: |
       The CA certificate to verify the server certificate with. This can be
@@ -40,44 +45,55 @@
       prefixed with "base64:". This config is only used if the message server
       URL given above starts with https.
     type: string
+    default:
   account-name:
     description: |
       The account this computer belongs to.
     type: string
+    default:
   registration-key:
     description: |
       The account-wide key used for registering clients.
     type: string
+    default:
   exchange-interval:
     description: |
       The number of seconds between server exchanges.
     type: int
+    default:
   urgent-exchange-interval:
     description: |
       The number of seconds between urgent server exchanges.
     type: int
+    default:
   ping-interval:
     description: |
       The number of seconds between pings.
     type: int
+    default:
   http-proxy:
     description: |
       The URL of the HTTP proxy, if one is needed.
     type: string
+    default:
   https-proxy:
     description: |
       The URL of the HTTPS proxy, if one is needed.
     type: string
+    default:
   tags:
     description: |
       Comma separated list of tag names to be sent to the server.
     type: string
+    default:
   script-users:
     description: |
       A comma-separated list of users to allow scripts to run.  To allow
       scripts to be run by any user, enter: ALL
     type: string
+    default:
   include-manager-plugins:
     description: |
       A comma-separated list of manager plugins to load.
     type: string
+    default:

=== modified file 'hooks/ceph.py'
--- hooks/ceph.py	2013-11-19 12:39:44 +0000
+++ hooks/ceph.py	2014-12-17 23:25:40 +0000
@@ -1,7 +1,7 @@
 import os
 from subprocess import check_output
 
-from common import LandscapeBroker, chown
+from common import chown
 
 
 CEPH_CLIENT_DIR = "ceph-client"

=== modified file 'hooks/install.py'
--- hooks/install.py	2014-08-28 12:13:31 +0000
+++ hooks/install.py	2014-12-17 23:25:40 +0000
@@ -46,7 +46,6 @@
             data_path])
 
 
-
 def build_from_launchpad(url):
     """The charm will install the code from the passed lp branch.
     """

=== modified file 'metadata.yaml'
--- metadata.yaml	2014-01-28 17:53:33 +0000
+++ metadata.yaml	2014-12-17 23:25:40 +0000
@@ -7,6 +7,7 @@
   account.  This package provides the Landscape client and requires a
   Landscape account.
 subordinate: true
+tags: [ ops, monitoring ]
 requires:
   container:
     interface: juju-info

=== added file 'tests/00-setup'
--- tests/00-setup	1970-01-01 00:00:00 +0000
+++ tests/00-setup	2014-12-17 23:25:40 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash -xe
+
+# Check if amulet is installed before adding repository and updating apt-get.
+if ! dpkg -s amulet; then
+    sudo add-apt-repository -y ppa:juju/stable
+    sudo apt-get update
+    sudo apt-get install -y amulet python-flake8 landscape-common
+fi
+
+# Install any additional python packages or software here.

=== removed file 'tests/001_install.test'
--- tests/001_install.test	2013-03-06 21:29:16 +0000
+++ tests/001_install.test	1970-01-01 00:00:00 +0000
@@ -1,95 +0,0 @@
-#!/bin/bash -eu
-
-source lib/test-helpers.sh
-
-test_setup() {
-    echo "INFO: Setup Test Environment"
-    juju deploy ubuntu
-    juju deploy --repository . local:landscape-client
-    juju add-relation ubuntu landscape-client
-}
-
-teardown() {
-    echo "INFO: Starting Teardown"
-    juju destroy-service ubuntu || /bin/true
-    juju destroy-service landscape-client || /bin/true
-}
-
-assert_command_on_unit() {
-    # Assert the command succeeds on the unit
-    # $1 = unit to contact
-    # $2 = command
-    if ! juju ssh $1 "$2"; then
-        error "CMD: $2 failed in $(diagnose)"
-        exit 1
-    fi
-}
-
-run_command_on_unit() {
-    juju ssh $1 "$2"
-}
-
-assert_config() {
-    # $1 = config
-    # $2 and on = list of strings to run through egrep
-    config=$1
-    shift
-    while [ $# -gt 0 ]; do
-        if ! echo "$config" | egrep "$1"; then
-            error "Config incorrect or missing: $1\n$(diagnose)"
-            exit 1
-        fi
-        shift
-    done
-}
-
-
-start-test "landscape-client add relation verify deploy"
-
-teardown
-test_setup
-
-# Ensure ubuntu started, and landscape-client is related without error
-jitsu watch --failfast \
-    ubuntu --state=started -r "ubuntu landscape-client" \
-    landscape-client --state=started
-
-
-# Ensure all ubuntu service units look as expected (unregistered)
-for i in $(jitsu get-service-info ubuntu public-address); do
-    unit=$(echo $i | cut -d: -f1)
-    host=$(echo $i | cut -d: -f2)
-    assert_command_on_unit $unit "ps -ef | grep -v /usr/bin/landscape-client"
-    assert_command_on_unit $unit "test -e /etc/landscape/client.conf"
-    assert_command_on_unit $unit "sudo cat /etc/landscape/client.conf"
-    assert_command_on_unit $unit "sudo grep computer_title /etc/landscape/client.conf | grep $unit"
-done
-
-# Set appropriate config for testing
-juju set --config lib/test-config.yaml landscape-client
-
-# We expect error since we aren't actually going to register the client
-jitsu watch \
-    landscape-client --state=configure-error
-
-# Ensure all ubuntu service units look as expected (registration-attempt)
-for i in $(jitsu get-service-info ubuntu public-address); do
-    unit=$(echo $i | cut -d: -f1)
-    host=$(echo $i | cut -d: -f2)
-    assert_command_on_unit $unit "ps -ef | grep -v /usr/bin/landscape-client"
-    assert_command_on_unit $unit "test -e /etc/landscape/client.conf"
-    config=$(run_command_on_unit $unit "sudo cat /etc/landscape/client.conf")
-    assert_config "$config" \
-        "ping_url = http://foo.example.com/ping" \
-        "data_path = /var/lib/landscape/client" \
-        "computer_title = $unit" \
-        "tags = foo,bar,baz" \
-        "registration_password = foo-registration-key" \
-        "url = https://foo.example.com/message-system" \
-        "include_manager_plugins = ScriptExecution" \
-        "script_users = ALL" \
-        "registration_key = foo-registration-key" \
-        "account_name = foo-account-name"
-done
-
-end-test "Deployed Successfully"

=== added file 'tests/10-deploy-register.py'
--- tests/10-deploy-register.py	1970-01-01 00:00:00 +0000
+++ tests/10-deploy-register.py	2014-12-17 23:25:40 +0000
@@ -0,0 +1,53 @@
+#!/usr/bin/env python3
+
+# This amulet test deploys the bundles.yaml file in this directory.
+
+import os
+import unittest
+import yaml
+import amulet
+import time
+
+seconds_to_wait = 1200
+
+
+class BundleTest(unittest.TestCase):
+    """Create a class for testing the charm in the unit test framework."""
+    @classmethod
+    def setUpClass(cls):
+        """Set up an amulet deployment using the bundle."""
+        origin = os.environ.get('LANDSCAPE_CLIENT_CHARM_ORIGIN', None)
+        d = amulet.Deployment()
+        bundle_path = os.path.join(os.path.dirname(__file__), 'bundles.yaml')
+        with open(bundle_path, 'r') as bundle_file:
+            contents = yaml.safe_load(bundle_file)
+            d.load(contents)
+        if origin is not None:
+            d.configure("landscape-client", {"origin": origin})
+        d.setup(seconds_to_wait)
+        d.sentry.wait(seconds_to_wait)
+        cls.d = d
+
+    def test_deployed(self):
+        """Test to see if the bundle deployed successfully."""
+        self.assertTrue(self.d.deployed)
+
+    def test_wait_for_success_in_log(self):
+        """landscape-client should successfully register."""
+        logfile = "/var/log/juju/unit-landscape-client*.log"
+        grep = "grep 'System successfully registered'"
+        # Only principal units are addressable in this way.
+        for i in range(20):
+            result = self.d.sentry.unit['ubuntu/0'].run(
+                "%s %s" % (grep, logfile))
+            if result[1] != 0:
+                time.sleep(10)
+            else:
+                # success, fall through
+                break
+        else:
+            raise AssertionError("Could not verify registration of the client")
+
+
+if __name__ == '__main__':
+    unittest.main()

=== removed file 'tests/Makefile'
--- tests/Makefile	2013-03-06 21:29:16 +0000
+++ tests/Makefile	1970-01-01 00:00:00 +0000
@@ -1,8 +0,0 @@
-build:
-	@mkdir -p precise
-	@test -e precise/landscape-client || ln -sf ../ precise/landscape-client
-	@echo "- Checking juju status, to make sure juju bootstrap has been done"
-	@juju status > /dev/null 2>/dev/null
-
-test:
-	./install-test

=== added file 'tests/bundles.yaml'
--- tests/bundles.yaml	1970-01-01 00:00:00 +0000
+++ tests/bundles.yaml	2014-12-17 23:25:40 +0000
@@ -0,0 +1,20 @@
+trusty:
+  series: trusty
+  services:
+    landscape-client:
+      # Amulet will replace this 'charm: landscape-client' with the branch
+      # that is currently under test on your local system.
+      charm: landscape-client
+      options:
+        account-name: charm-testing
+        registration-key: testsaregood
+        tags: testing,foo,tag
+        ping-url: http://staging.landscape.canonical.com/ping
+        url: https://staging.landscape.canonical.com/message-system
+
+    ubuntu:
+      charm: cs:trusty/ubuntu
+      num_units: 1
+
+  relations:
+    - [ "landscape-client", "ubuntu" ]

=== removed directory 'tests/lib'
=== removed file 'tests/lib/test-config.yaml'
--- tests/lib/test-config.yaml	2013-03-06 21:29:16 +0000
+++ tests/lib/test-config.yaml	1970-01-01 00:00:00 +0000
@@ -1,8 +0,0 @@
-landscape-client:
-    account-name: foo-account-name
-    registration-key: foo-registration-key
-    tags: foo,bar,baz
-    ping-url: http://foo.example.com/ping
-    url: https://foo.example.com/message-system
-    script-users: ALL
-    include-manager-plugins: ScriptExecution

=== removed file 'tests/lib/test-helpers.sh'
--- tests/lib/test-helpers.sh	2013-04-26 21:21:18 +0000
+++ tests/lib/test-helpers.sh	1970-01-01 00:00:00 +0000
@@ -1,114 +0,0 @@
-#!/bin/bash
-
-set -eu
-
-# Test setup
-readonly TEST_NAME=$0
-
-# Test counts. Neither skips or errors are necessarily fatal.
-SKIPS=0
-ERRORS=0
-UNHANDLED=0
-
-
-function info() {
-    echo "INFO: $@"
-}
-
-function ok() {
-    echo "PASS: $@"
-}
-
-function skip() {
-    echo "SKIP: $@"
-    (( SKIPS +=1 ))
-}
-
-function error() {
-    echo "FAIL: $@"
-    (( ERRORS += 1 ))
-}
-
-
-function start-test() {
-    readonly DATADIR=$(mktemp -d --tmpdir "${TEST_NAME}.XXXXXXX")
-    info "Executing ${TEST_NAME} with results in $DATADIR: $1"
-    setup
-    trap trapped-teardown EXIT TERM INT
-}
-
-function setup() {
-    # Empty, override in actual test as needed
-    true
-}
-
-
-function end-test() {
-    info "Completed test $TEST_NAME"
-    trap - EXIT
-    test-teardown
-}
-
-
-function teardown() {
-    # Empty, override in actual test as needed
-    true
-}
-
-
-function test-teardown() {
-    teardown
-
-    if [[ "$ERRORS" -gt 0 ]] ; then
-        TEST_RESULT=1
-    elif [[ "$SKIPS" -gt 0 ]] ; then
-        TEST_RESULT=100
-        touch $DATADIR/passed
-    else 
-        TEST_RESULT=0
-        touch $DATADIR/passed
-    fi
-
-    if [ -n "$DATADIR" ] ; then
-        if [ -f $DATADIR/passed ]; then
-            info "Passed test results for ${TEST_NAME} in ${DATADIR}, skips=${SKIPS}"
-            # rm -r $DATADIR
-        else
-            info "Failed test results for ${TEST_NAME} in ${DATADIR}, errors=${ERRORS}, skips=${SKIPS}"
-            if [ -f $DATADIR/wget.log ] ; then
-                info "BEGIN wget.log"
-                cat $DATADIR/wget.log
-                info "END wget.log"
-            fi
-        fi
-    fi
-    exit $TEST_RESULT
-}
-
-
-function diagnose() {
-    # Unpack the caller info to get some useful diagnostics for
-    # tracing any assertion issues
-    local data=( $(caller 1) )
-    local lineno=${data[0]}
-    local function_name=${data[1]}
-    local script_name=${data[2]}
-    echo "function $function_name at line $lineno ($script_name)"
-}
-
-function trapped-teardown() {
-    (( UNHANDLED += 1 ))
-    error "Untrapped error in $(diagnose)"
-    test-teardown
-}
-
-function assert() {
-    # The assert function takes two args, such as:
-    # assert "what we are asserting" "non empty result passes"
-    # If the second arg is empty or undefined, the assertion fails
-    if [[ "$#" -lt 2 || -z "$2" ]] ; then
-        error "$1, assertion failed in $(diagnose)"
-    else
-        ok "$1"
-    fi
-}

=== removed directory 'tests/precise'
=== removed symlink 'tests/precise/landscape-client'
=== target was u'../..'
=== removed file 'tests/test.sh'
--- tests/test.sh	2013-04-26 21:24:37 +0000
+++ tests/test.sh	1970-01-01 00:00:00 +0000
@@ -1,6 +0,0 @@
-#!/bin/bash -e
-
-DIR="$( cd "$( dirname "$0" )" && pwd )"
-cd $DIR
-
-run-parts -v --exit-on-error --regex '^.*\.test$' .

=== added file 'tests/test.yaml'
--- tests/test.yaml	1970-01-01 00:00:00 +0000
+++ tests/test.yaml	2014-12-17 23:25:40 +0000
@@ -0,0 +1,7 @@
+packages:
+    - python-psutil
+    - amulet
+    - python3-amulet
+    - python-amulet
+    - landscape-common
+    - python-flake8