Merge lp:~clint-fewbar/charm-tools/charm-tests-spec into lp:~charmers/charm-tools/trunk

[Clint Byrum]

Fixes per community feedback.

[lp:~clint-fewbar/charm-tools/charm-tests-spec]

128. By Clint Byrum on 2012-01-31

updating example to reflect output guidelines

129. By Clint Byrum on 2012-02-01

Explain what happens to services left behind after tests exit

[Mark Mims (mark-mims)]

just grabbed this to learn more lp...

for the record this is in lp:juju/docs now

review: Disapprove

Unmerged revisions

129. By Clint Byrum on 2012-02-01

Explain what happens to services left behind after tests exit

128. By Clint Byrum on 2012-01-31

updating example to reflect output guidelines

127. By Clint Byrum on 2012-01-31

clarify requirements.yaml and move get-unit-info tip into footnotes

126. By Clint Byrum on 2012-01-31

rearranging examples and clarifying exit codes

=== modified file 'doc/source/charm-tests.rst'
--- doc/source/charm-tests.rst	2012-01-26 01:24:15 +0000
+++ doc/source/charm-tests.rst	2012-02-01 21:14:18 +0000
@@ -80,10 +80,11 @@
 * Internet access will be restricted from the testing host.
 
 If present, tests/requirements.yaml will be read to determine packages
-that need to be installed in order to facilitate the tests. The packages
-can *only* be installed from the official, default Ubuntu archive for the
-release which the charm is intended for. The format of requirements.yaml
-is as such::
+that need to be installed on the host running tests in order to facilitate
+the tests. The packages can *only* be installed from the official,
+default Ubuntu archive for the release which the charm is intended for,
+from any of the repositories enabled by default in said release. The
+format of requirements.yaml is as such::
 
     packages: [ package1, package2, package3 ]
 
@@ -92,35 +93,52 @@
 as the file which contains it does not end in ``.test``. Note that build
 tools cannot be assumed to be available on the testing system.
 
-Test Runner
-===========
-
-A test runner will periodically poll the collection of charms for changes
-since the last test run. If there have been changes, the entire set of
-changes will be tested as one delta. This delta will be recorded in the
-test results in such a way where a developer can repeat the exact set
-of changes for debugging purposes.
-
-All of the charms will be scanned for tests in lexical order by
-series, charm name, branch name. Non official charms which have not
-been reviewed by charmers will not have their tests run until the test
-runner's restrictions have been vetted for security, since we will be
-running potentially malicious code. It is left to the implementor to
-determine what mix of juju, client platform, and environment settings
-are appropriate, as all of these are variables that will affect the
-running charms, and so may affect the outcome.
-
-Example
-=======
+Purpose of tests
+----------------
+
+The purpose of these tests is to assert that the charm works well on the
+intended platform and performs the expected configuration steps. Examples
+of things to test in each charm beyond install/start is:
+
+* After install, expose, and adding of required relations, the service is
+  listening on the intended ports and is functional.
+* Adding, removing, and re-adding a relation should work without error.
+* Setting config values should result in the config value reflected in
+  the service's configuraion.
+* Adding multiple units to a web app charm and relating to a load balancer
+  results in the same HTML on both units directly and the load balancer.
+
+Exit Codes
+----------
+
+Upon exit, the test's exit code will be evaluated to mean the following:
+
+* 0:   Test passed
+* 1:   Failed test
+* 100: Test is skipped because of incomplete environment
+
+Output
+------
+
+There is a general convention which output should follow, though it will
+not be interpreted by machine. On stdout, a message indicating the reason
+for the exit code should be printed, with a prefix string corresponding to
+the exit codes defined above. The correlation is:
+
+* PASS - 0
+* FAIL - 1
+* SKIP - 100
+
+Example Tests
+-------------
 
 Deploy requirements and Poll
-----------------------------
-
-The following example test script uses a tool that is not widely available
-yet, ``get-unit-info``. In the future enhancements should be made to
-juju core to allow such things to be made into plugins. Until then,
-it can be included in each test dir that uses it, or we can build a
-package of tools that are common to tests.::
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The test below [*]_ deploys mediawiki with mysql and memcached related to it,
+and then tests to make sure it returns a page via http with "<title>"
+somewhere in the content.::
+
 
     #!/bin/sh
 
@@ -158,7 +176,7 @@
     done
 
     if [ -z "$host" ] ; then
-        echo ERROR: status timed out 
+        echo FAIL: status timed out 
         exit 1
     fi
 
@@ -168,7 +186,7 @@
     wget --tries=100 --timeout=6 http://$host/ -O - -a $datadir/wget.log | grep -q '<title>'
 
     if [ $try -eq 600 ] ; then
-        echo ERROR: Timed out waiting.
+        echo FAIL: Timed out waiting.
         exit 1
     fi
 
@@ -180,3 +198,99 @@
     echo INFO: PASS
     exit 0
 
+Test config settings
+~~~~~~~~~~~~~~~~~~~~
+
+The following example tests checks to see if the default_port change
+the admin asks for is actually respected post-deploy::
+
+    #!/bin/sh
+
+    if [ -z "`which nc`" ] ; then
+        echo "SKIP: cannot run tests without netcat"
+        exit 100
+    fi
+
+    set -e
+
+    teardown() {
+        juju destroy-service mongodb
+    }
+    trap teardown EXIT
+
+    juju deploy mongodb
+    juju expose mongodb
+
+    for try in `seq 1 600` ; do
+        host=`juju status | tests/get-unit-info mongodb public-address`
+        if [ -z "$host" ] ; then
+            sleep 1
+        else
+            break
+        fi
+    done
+
+    if [ -z "$host" ] ; then
+        echo FAIL: status timed out
+        exit 1
+    fi
+
+    assert_is_listening() {
+        local port=$1
+        listening=""
+        for try in `seq 1 10` ; do
+            if ! nc $host $port < /dev/null ; then
+                continue
+            fi
+            listening="$port"
+            break
+        done
+
+        if [ -z "$listening" ] ; then
+           echo "FAIL: not listening on port $port after 10 retries"
+           return 1
+        else
+           echo "PASS: listening on port $listening"
+           return 0
+        fi
+    }
+
+    assert_is_listening 27017
+
+    juju set mongodb default_port=55555
+
+    assert_is_listening 55555
+    echo PASS: config change tests passed.
+    exit 0
+
+.. [*] get-unit-info
+   The example tests script uses a tool that is not widely available yet,
+   ``get-unit-info``. In the future enhancements should be made to juju
+   core to allow such things to be made into plugins. Until then, it can
+   be included in each test dir that uses it, or we can build a package
+   of tools that are common to tests.
+
+Test Runner
+===========
+
+A test runner will periodically poll the collection of charms for changes
+since the last test run. If there have been changes, the entire set of
+changes will be tested as one delta. This delta will be recorded in the
+test results in such a way where a developer can repeat the exact set
+of changes for debugging purposes.
+
+All of the charms will be scanned for tests in lexical order by
+series, charm name, branch name. Non official charms which have not
+been reviewed by charmers will not have their tests run until the test
+runner's restrictions have been vetted for security, since we will be
+running potentially malicious code. It is left to the implementor to
+determine what mix of juju, client platform, and environment settings
+are appropriate, as all of these are variables that will affect the
+running charms, and so may affect the outcome.
+
+If tests exit with services still in the environment, the test runner
+may clean them up, whether by destroying the environment or destroying
+the services explicitly, and the machines may be terminated as well.
+Tests should cleanup any services they start so that the test script
+is more indpendent and idempotent. Any artifacts needed from the test
+machines should be retrieved and displayed before the test exits.