Merge lp:~clint-fewbar/pyjuju/charm-tests-spec into lp:~juju/pyjuju/docs

[Clint Byrum]

This is a draft specification for implementing large scale charm tests.

This is a draft specification for implementing large scale charm tests.

https://codereview.appspot.com/5624044/

[Gustavo Niemeyer (niemeyer)]

Thanks Clint. Here are some comments:

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst
File source/charm-tests.rst (right):

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode51
source/charm-tests.rst:51: utilized to attach tests to charms. Under the
charm root directory,
"To facilitate tests attached to charms .. to attach tests to charms."

The sentence is a bit self-referential. Would be nice to reword

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode60
source/charm-tests.rst:60: * the default environment is bootstrapped
To be more clear, we might say something like:

* A testing environment is already bootstrapped and is made the default
for command line usage.

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode61
source/charm-tests.rst:61: * The CWD is the charm root directory
I think this should be the "tests/" directory itself. Note that the test
doesn't actually have a strong relation to the content of the charm. The
content is deployed into the remote machine, and the test is sitting
locally. Its interaction will be mainly remote.

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode64
source/charm-tests.rst:64: charm url and/or repository arguments of the
test runner's choice. This
We have to qualify a bit better what the "runner's choice" mean, since a
charmer reading that will feel like "Okay, but _what is the version_?".

Maybe something like "the current version at the moment the test is
run"? This gives us some freedom to move back and forth, and gives the
charmer a vague feeling of what to expect.

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode69
source/charm-tests.rst:69: * a special sub-command of juju,
``deploy-previous``, will deploy the
This command doesn't exist. Would you mind to drop the upgrade aspect
from the specification entirely? I'd be happy to debate about a follow
up improvement once we have the basics in place.

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode76
source/charm-tests.rst:76: * ``~/.juju`` will not be accessible to the
tests
Read-only, maybe? It needs to be accessible for reading purposes or the
command doesn't work.

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode82
source/charm-tests.rst:82: If present, tests/requirements.yaml will be
read to determine packages
Looks good, but can we please name that as "tests/tests.yaml", so that
this may be used for other needs too, when/if they show up?

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode90
source/charm-tests.rst:90: If a tool is needed to perform a test and not
available in the Ubuntu
s/and not/and is not/?

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode111
source/charm-tests.rst:111: running charms, and so may affect the
outcome.
Section looks great.

https://codereview.appspot.com/5624044/diff/1/source/charm-tests.rst#newcode141
source/charm-tests.rst:141: trap teardown EXIT
Destroying the temporary state manually should not be necessary. The
harness should be able to put the...

[lp:~clint-fewbar/pyjuju/charm-tests-spec]

6. By Clint Byrum on 2012-02-06

Rewording paragraph to be more succinct.

7. By Clint Byrum on 2012-02-06

clarifying default environment

8. By Clint Byrum on 2012-02-06

moving CWD to tests directory to encourage better encapsulation of test code

9. By Clint Byrum on 2012-02-06

using strong language in explanation of charm resolver

10. By Clint Byrum on 2012-02-06

dropping deploy-previous pending some discussion of how best to test upgrade-charm

11. By Clint Byrum on 2012-02-06

No need to state this, as it complicates things unnecessarily and is implied by not guaranteeing access to it

12. By Clint Byrum on 2012-02-06

renaming requirements.yaml to tests.yaml to allow broader usage in the future

13. By Clint Byrum on 2012-02-06

grammar is our friend

14. By Clint Byrum on 2012-02-06

Re-merging fixes from email discussion. Moves test runner to the end, adds another example, clarifies exit codes and intended test output.

15. By Clint Byrum on 2012-02-23

Removing suggestions to cleanup services. Also clarifying output suggestions.

[Mark Mims (mark-mims)]

review: Approve

[Mark Mims (mark-mims)]

merged into lp:~charm-contributors/juju/docs instead

=== added file 'source/charm-tests.rst'
--- source/charm-tests.rst	1970-01-01 00:00:00 +0000
+++ source/charm-tests.rst	2012-02-23 22:37:12 +0000
@@ -0,0 +1,289 @@
+==============
+Charm Testing
+==============
+
+Intro
+=====
+
+**DRAFT**
+
+Juju has been designed from the start to foster a large collection of
+"charms". Charms are expected to number in the thousands, and be self
+contained, with well defined interfaces for defining their relationships
+to one another.
+
+Because this is a large complex system, not unlike a Linux software
+distribution, there is a need to test the charms and how they interact
+with one another. This specification  defines a plan for implementing
+a simple framework to help this happen.
+
+Static tests have already been implemented in the ``charm proof`` command
+as part of ``charm-tools``. Any static testing of charms is beyond the
+scope of this specification.
+
+Phase 1 - Generic tests
+=======================
+
+All charms share some of the same characteristics. They all have a
+yaml file called ``metadata.yaml``, and when deployed, juju will always
+attempt to progress the state of the service from install to config to
+started. Because of this, all charms can be tested using the following
+algorithm::
+
+ deploy charm
+ while state != started
+   if timeout is reached, FAIL
+   if state == install_error, config_error, or start_error, FAIL
+   if state == started, PASS
+
+Other generic tests may be identified, so a collection of generic tests should be the focus of an implementation.
+
+Note that this requirement is already satisfied by Mark Mims' jenkins tester:
+http://charmtests.markmims.com/
+
+Phase 2 - Charm Specific tests
+==============================
+
+Charm authors will have the best insight into whether or not a charm is
+working properly.
+
+A simple structure will be utilized to attach tests to charms. Under the
+charm root directory, a sub-directory named 'tests' will be scanned by
+a test runner for executable files matching the glob ``*.test``. These
+will be run in lexical order by the test runner, with a predictible
+environment. The tests can make the following assumptions:
+
+* A minimal install of the release of Ubuntu which the charm is targetted
+  at will be available.
+* A version of juju is installed and available in the system path.
+* A juju environment with no services deployed inside it is already
+  bootstrapped, and will be the default for command line usage.
+* The CWD is the ``tests`` directory off the charm root.
+* Full network access to deployed nodes will be allowed.
+* the bare name of any charm in arguments to juju will be resolved to a
+  charm url and/or repository arguments of the test runner's choice. This
+  means that if you need mysql, you do not do ``juju deploy cs:mysql`` or
+  ``juju deploy --repository ~/charms local:mysql``, but just ``juju deploy
+  mysql``. A wrapper will resolve this to the latest version of the
+  given charm from the list of official charms.
+
+The following restrictions may be enforced:
+
+* Internet access will be restricted from the testing host.
+
+If present, tests/tests.yaml will be read to determine packages
+that need to be installed on the host running tests in order to facilitate
+the tests. The packages can *only* be installed from the official,
+default Ubuntu archive for the release which the charm is intended for,
+from any of the repositories enabled by default in said release. The
+format of tests.yaml is as such::
+
+    packages: [ package1, package2, package3 ]
+
+If a tool is needed to perform a test and is not available in the Ubuntu
+archive, it can also be included in the ``tests/`` directory, as long
+as the file which contains it does not end in ``.test``. Note that build
+tools cannot be assumed to be available on the testing system.
+
+Purpose of tests
+----------------
+
+The purpose of these tests is to assert that the charm works well on the
+intended platform and performs the expected configuration steps. Examples
+of things to test in each charm beyond install/start is:
+
+* After install, expose, and adding of required relations, the service is
+  listening on the intended ports and is functional.
+* Adding, removing, and re-adding a relation should work without error.
+* Setting config values should result in the config value reflected in
+  the service's configuraion.
+* Adding multiple units to a web app charm and relating to a load balancer
+  results in the same HTML on both units directly and the load balancer.
+
+Exit Codes
+----------
+
+Upon exit, the test's exit code will be evaluated to mean the following:
+
+* 0:   Test passed
+* 1:   Failed test
+* 100: Test is skipped because of incomplete environment
+
+Output
+------
+
+There is a general convention which output should follow, though it will
+not be interpreted by machine. On stdout, a message indicating the reason
+for the exit code should be printed, with a prefix string corresponding to
+the exit codes defined above. The correlation is:
+
+* PASS - 0
+* FAIL - 1
+* SKIP - 100
+
+Anything else intentional should be prefixed with the word 'INFO'. If the
+contents of files are to be logged, the contents should be preceeded by
+``INFO: BEGIN filename``, where filename is a logical name unique to
+this run of the test, and then the file ended with ``INFO: END filename``.
+
+Example Tests
+-------------
+
+Deploy requirements and Poll
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The test below [*]_ deploys mediawiki with mysql and memcached related to it,
+and then tests to make sure it returns a page via http with "<title>"
+somewhere in the content.::
+
+
+    #!/bin/sh
+
+    set -e
+
+    teardown() {
+        if [ -n "$datadir" ] ; then
+            if [ -f $datadir/passed ]; then
+                rm -r $datadir
+            else
+                echo INFO: $datadir preserved
+                if [ -f $datadir/wget.log ] ; then
+                    echo INFO: BEGIN wget.log
+                    cat $datadir/wget.log
+                    echo INFO: END wget.log
+                fi
+            fi
+        fi
+    }
+    trap teardown EXIT
+
+
+    juju deploy mediawiki
+    juju deploy mysql
+    juju deploy memcached
+    juju add-relation mediawiki:db mysql:db
+    juju add-relation memcached mediawiki
+    juju expose mediawiki
+
+    for try in `seq 1 600` ; do
+        host=`juju status | tests/get-unit-info mediawiki public-address`
+        if [ -z "$host" ] ; then
+            sleep 1 
+        else
+            break
+        fi
+    done
+
+    if [ -z "$host" ] ; then
+        echo FAIL: status timed out 
+        exit 1
+    fi
+
+    datadir=`mktemp -d ${TMPDIR:-/tmp}/wget.test.XXXXXXX`
+    echo INFO: datadir=$datadir
+
+    wget --tries=100 --timeout=6 http://$host/ -O - -a $datadir/wget.log | grep -q '<title>'
+
+    if [ $try -eq 600 ] ; then
+        echo FAIL: Timed out waiting.
+        exit 1
+    fi
+
+    touch $datadir/passed
+
+    trap - EXIT
+    teardown
+
+    echo PASS
+    exit 0
+
+Test config settings
+~~~~~~~~~~~~~~~~~~~~
+
+The following example tests checks to see if the default_port change
+the admin asks for is actually respected post-deploy::
+
+    #!/bin/sh
+
+    if [ -z "`which nc`" ] ; then
+        echo "SKIP: cannot run tests without netcat"
+        exit 100
+    fi
+
+    set -e
+
+    juju deploy mongodb
+    juju expose mongodb
+
+    for try in `seq 1 600` ; do
+        host=`juju status | tests/get-unit-info mongodb public-address`
+        if [ -z "$host" ] ; then
+            sleep 1
+        else
+            break
+        fi
+    done
+
+    if [ -z "$host" ] ; then
+        echo FAIL: status timed out
+        exit 1
+    fi
+
+    assert_is_listening() {
+        local port=$1
+        listening=""
+        for try in `seq 1 10` ; do
+            if ! nc $host $port < /dev/null ; then
+                continue
+            fi
+            listening="$port"
+            break
+        done
+
+        if [ -z "$listening" ] ; then
+           echo "FAIL: not listening on port $port after 10 retries"
+           return 1
+        else
+           echo "PASS: listening on port $listening"
+           return 0
+        fi
+    }
+
+    assert_is_listening 27017
+
+    juju set mongodb default_port=55555
+
+    assert_is_listening 55555
+    echo PASS: config change tests passed.
+    exit 0
+
+.. [*] get-unit-info
+   The example tests script uses a tool that is not widely available yet,
+   ``get-unit-info``. In the future enhancements should be made to juju
+   core to allow such things to be made into plugins. Until then, it can
+   be included in each test dir that uses it, or we can build a package
+   of tools that are common to tests.
+
+Test Runner
+===========
+
+A test runner will periodically poll the collection of charms for changes
+since the last test run. If there have been changes, the entire set of
+changes will be tested as one delta. This delta will be recorded in the
+test results in such a way where a developer can repeat the exact set
+of changes for debugging purposes.
+
+All of the charms will be scanned for tests in lexical order by
+series, charm name, branch name. Non official charms which have not
+been reviewed by charmers will not have their tests run until the test
+runner's restrictions have been vetted for security, since we will be
+running potentially malicious code. It is left to the implementor to
+determine what mix of juju, client platform, and environment settings
+are appropriate, as all of these are variables that will affect the
+running charms, and so may affect the outcome.
+
+If tests exit with services still in the environment, the test runner
+may clean them up, whether by destroying the environment or destroying
+the services explicitly, and the machines may be terminated as well.
+Any artifacts needed from the test machines should be retrieved and
+displayed before the test exits.