Merge lp:~codehelp/lava-dispatcher/multinode-docs-update into lp:lava-dispatcher/multinode

[Neil Williams]

This is the update containing the docs as shown in the demo.

[Milo Casagrande (milo)]

It looked good on the demo, and going through it I do not see anything broken or wrong.
So, +1 from me.

review: Approve

=== added file 'doc/debugging.rst'
--- doc/debugging.rst	1970-01-01 00:00:00 +0000
+++ doc/debugging.rst	2013-08-23 10:36:27 +0000
@@ -0,0 +1,119 @@
+.. _debugging:
+
+Debugging LAVA test definitions
+*******************************
+
+.. _singlenode:
+
+Convert Multi-Node jobs to single node
+======================================
+
+The scripts available in the :ref:`multinode_api` are not installed for
+test jobs which are not part of a MultiNode group, so the job will simply
+fail that test as a ``command not found``.
+
+Therefore, by reversing the :ref:`changes_to_json`, a MultiNode JSON file
+can be converted to singlenode.
+
+Other calls which may require communication with other devices may need
+to be removed from your YAML. This can be extended to retain a set of
+singlenode YAML files in which new wrapper scripts and new builds are
+tested.
+
+The Job Definition of one job within a MultiNode group may be a good
+starting point for creating a singlenode equivalent.
+
+.. _set_x:
+
+Always use set -x in wrapper scripts
+====================================
+
+By viewing the complete log, the complete processing of the wrapper script
+becomes obvious.
+
+::
+
+ #!/bin/sh
+ set -e
+ set -x
+
+.. _shell_operators:
+
+Avoid using shell operators in YAML lines
+=========================================
+
+Pipes, redirects and nested sub shells will not work reliably when put
+directly into the YAML. Use a wrapper script (with :ref:`set -x <set_x>`).
+
+::
+
+ #!/bin/sh
+
+ set -e
+ set -x
+ ifconfig|grep "inet addr"|grep -v "127.0.0.1"|cut -d: -f2|cut -d' ' -f1
+
+Un-nested sub-shells do work::
+
+ - lava-test-case multinode-send-network --shell lava-send network hostname=$(hostname) fqdn=$(hostname -f)
+
+.. _check_messageid:
+
+Check that your message ID labels are consistent
+================================================
+
+A :ref:`lava_wait` must be preceded by a :ref:`lava_send` from at least
+one other device in the group or the waiting device will :ref:`timeout <timeouts>`
+
+This can be a particular problem if you remove test definitions from the
+JSON or edit a YAML file without checking other uses of the same file.
+
+``#`` can be used as a comment in YAML but JSON does not support
+comments, so take care.
+
+.. _parsers:
+
+Test your result parsers
+========================
+
+If the YAML uses a custom result parser, configure one of your YAML files
+to output the entire test result output to stdout so that you can
+reliably capture a representative block of output. Test your proposed
+result parser against the block using your favourite language.
+
+Comment out the parser from the YAML if there are particular problems,
+just to see what the default LAVA parsers can provide.
+
+.. _paths:
+
+Be obsessive about paths and scripts
+====================================
+
+* If you use ``cd`` in your YAML, always store where you were and where you end up using ``pwd``. 
+* Output your location prior to calling local wrapper scripts.
+* Ensure that all wrapper scripts are executable in your VCS
+* Ensure that the relevant interpreter is installed. e.g. python is not necessarily part of the test image.
+* Consider installing ``realpath`` and use that to debug your directory structure.
+  * Avoid the temptation of using absolute paths - LAVA may need to change the absolute locations.
+
+.. _failed_tests:
+
+A failed test is not necessarily a bug in the test
+==================================================
+
+Always check whether the test result came back as failed due to some
+cause other than the test definition itself. Particularly with MultiNode,
+a test result can fail due to some problem on a different board within
+the group.
+
+.. _json_files:
+
+Check your JSON files
+=====================
+
+Syntax problems will be picked up by LAVA when you submit but also check
+that the URLs listed in the JSON are correct. Keep your YAML descriptions,
+names and filenames unique so that it is easier to pick up if the JSON
+simply calls the wrong YAML test definition.
+
+
 
=== modified file 'doc/multinode.rst'
--- doc/multinode.rst	2013-08-19 10:36:07 +0000
+++ doc/multinode.rst	2013-08-23 10:36:27 +0000
@@ -29,6 +29,7 @@
 
    multinodeapi.rst
    multinode-usecases.rst
+   debugging.rst
 
 Hardware requirements and virtualisation
 ****************************************
@@ -70,6 +71,8 @@
 If more than one, but not all, roles share one particular action, that action will need to be repeated
 within the JSON file, once for each role using that action.
 
+.. _changes_to_json:
+
 Changes to submission JSON
 ==========================
 
@@ -159,6 +162,8 @@
 or a lava-wait will fail on any node which waits longer than the default timeout. The node will receive a failure
 response.
 
+.. _timeouts:
+
 Recommendations on timeouts
 ===========================
 
@@ -178,6 +183,52 @@
 waiting jobs have already failed due to a problem elsewhere in the group. If timeouts are too short,
 jobs will fail unnecessarily.
 
+Balancing timeouts
+^^^^^^^^^^^^^^^^^^
+
+Individual actions and commands can have differing timeouts, so avoid the temptation to change the
+default timeout when a particular action times out in a Multi-Node job. If a particular ``lava-test-shell``
+takes a long time, set an explicit timeout for that particular action:
+
+::
+
+ {
+    "timeout": 900,
+    "job_name": "netperf multinode tests",
+    "logging_level": "DEBUG",
+ }
+
+
+::
+
+        {
+            "command": "lava_test_shell",
+            "parameters": {
+                "testdef_repos": [
+                    {
+                        "git-repo": "git://git.linaro.org/people/guoqing.zhu/netperf-multinode.git",
+                        "testdef": "netperf-multinode-c-network.yaml"
+                    }
+                ],
+                "timeout": 2400,
+                "role": "client"
+            }
+        },
+        {
+            "command": "lava_test_shell",
+            "parameters": {
+                "testdef_repos": [
+                    {
+                        "git-repo": "git://git.linaro.org/people/guoqing.zhu/netperf-multinode.git",
+                        "testdef": "netperf-multinode-s-network.yaml"
+                    }
+                ],
+                "timeout": 1800,
+                "role": "server"
+            }
+        },
+
+
 Running a server on the device-under-test
 *****************************************
 
 
=== modified file 'doc/multinodeapi.rst'
--- doc/multinodeapi.rst	2013-08-19 10:36:07 +0000
+++ doc/multinodeapi.rst	2013-08-23 10:36:27 +0000
@@ -1,3 +1,5 @@
+.. _multinode_api:
+
 MultiNode API
 =============
 
@@ -6,6 +8,8 @@
 definitions which need to transfer files, long messages or other large amounts of data need to set up their
 own network configuration, access and download methods and do the transfer in the test definition.
 
+.. _lava_self:
+
 lava-self
 ---------
 
@@ -13,6 +17,8 @@
 
 Usage: ``lava-self``
 
+.. _lava_role:
+
 lava-role
 ---------
 
@@ -25,6 +31,8 @@
 
     $ ./run-$(lava-role)
 
+.. _lava-group:
+
 lava-group
 ----------
 
@@ -42,6 +50,8 @@
     highbank02  backend
     highbank03  backend
 
+.. _lava_send:
+
 lava-send
 ---------
 
@@ -55,6 +65,8 @@
 Examples will be provided below, together with ``lava-wait`` and
 ``lava-wait-all``.
 
+.. _lava_wait:
+
 lava-wait
 ---------
 
@@ -75,6 +87,8 @@
 will be returned by subsequent calls to ``lava-wait`` for that message
 ID. Use a different message ID to collate different message data.
 
+.. _lava_wait_all:
+
 lava-wait-all
 -------------
 
@@ -108,6 +122,8 @@
 As with ``lava-wait``, the message ID is persistent for the duration of
 the MultiNode group.
 
+.. _lava_sync:
+
 lava-sync
 ---------
 
@@ -119,6 +135,8 @@
 ``lava-sync foo`` is effectively the same as ``lava-send foo`` followed
 by ``lava-wait-all foo``.
 
+.. _lava_network:
+
 lava-network
 ------------
 
 
=== modified file 'doc/usecaseone.rst'
--- doc/usecaseone.rst	2013-08-19 10:36:07 +0000
+++ doc/usecaseone.rst	2013-08-23 10:36:27 +0000
@@ -1,34 +1,39 @@
+.. _use_case_one:
+
 Use Case One - Setting up a simple client:server test definition.
 *****************************************************************
 
-One device needs to obtain / prepare some data and then make the data 
+One device needs to obtain / prepare some data and then make the data
 available to another device in the same group.
 
 Source Code
 ===========
 
- * The YAML snippets in this example are not complete, for a working example of the code, see:
+* The YAML snippets in this example are not complete, for a working example of the code, see:
 
   https://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=forwarder.yaml;hb=refs/heads/master
 
   https://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=receiver.yaml;hb=refs/heads/master
 
-  https://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=json/kvm-beagleblack-group.json;hb=HEAD
+  https://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=json/beagleblack-use-case.json;hb=HEAD
 
 Requirements
 ============
 
- * A mechanism to obtain the data, presumably from some third-party source
- * A sync to ensure that the file is ready to be offered to the other device
-   * This ensures that the attempt to receive does not start early
- * A message to the original board that the data has been received and verified
-   * This ensures that any cleanup of the data does not happen before the transfer is complete.
+1. A mechanism to obtain the data, presumably from some third-party source
+2. A sync to ensure that the file is ready to be offered to the other device
+
+ 2.1. This ensures that the attempt to receive does not start early
+
+3. A message to the original board that the data has been received and verified
+
+ 3.1. This ensures that any cleanup of the data does not happen before the transfer is complete.
 
 Methods
 =======
 
- * Install a package which can obtain the data from the third party source
- * Install a package which can provide the means to get the data to the other board
+* Install a package which can obtain the data from the third party source
+* Install a package which can provide the means to get the data to the other board
 
 Control flow
 ============
@@ -66,8 +71,8 @@
 of the filter. To start each YAML file, ensure that the metadata contains
 two metadata fields:
 
- * format - **Lava-Test Test Definition 1.0**
- * description - your own descriptive text
+* format : **Lava-Test Test Definition 1.0**
+* description : your own descriptive text
 
 It is useful to also add the maintainer field with your email address
 as this will be needed later if the test is to be added to one of the
@@ -82,10 +87,50 @@
     maintainer:
         - [e-mail]
 
+Installing packages for use in a test
+-------------------------------------
+
+If your test image raises a usable network interface by default on boot,
+the YAML can specify a list of packages which need to be installed for
+this test definition:
+
+::
+
+ install:
+    deps:
+        - wget
+        - apache2
+
+If your test needs to raise the network interface itself, the package
+installation will need to be done in the run steps::
+
+ run:
+    steps:
+        - lava-test-case linux-linaro-ubuntu-route-ifconfig-up --shell ifconfig eth0 up
+        - lava-test-case apt-update --shell apt-get update
+        - lava-test-case install-deps --shell apt-get -y install wget apache2
+
+Note that although KVM devices can use apt, the network interface fails
+the LAVA test, so use the manual install steps for non-bridged KVM devices.
 
 Preparing the test to send data
 -------------------------------
 
+``modify-data.sh`` would, presumably, unpack the data, modify it in
+some way and pack it back up again. In this example, it would be a no-op
+but note that it still needs to exist in the top level directory of your
+VCS repo and be executable.
+
+Any packages required by ``modify-data.sh`` need to be added to the install
+deps of sender.yaml. Providing useful contents of ``modify-data.sh`` is
+left as an exercise for the reader.
+
+Modification happens before the :ref:`lava_sync` ``download`` which tells the
+receiver that the data is ready to be transferred.
+
+The sender then waits for the receiver to acknowledge a correct download
+using :ref:`lava_sync` ``received`` and cleans up.
+
 sender.yaml
 ^^^^^^^^^^^
 
@@ -98,7 +143,6 @@
 
  run:
    steps:
-        - lava-test-case linux-linaro-ubuntu-route-ifconfig-up --shell ifconfig eth0 up
         - lava-test-case multinode-network --shell lava-network broadcast eth0
         - lava-test-case wget-file --shell wget -O /var/www/testfile http://releases.linaro.org/latest/android/arndale/userdata.tar.bz2
         - ./modify-data.sh
@@ -112,12 +156,10 @@
 The receiver needs to know where to find the data. The sender can ensure that the
 file is in a particular location, it is up to the YAML to get the rest of the
 information of the network address of the sender. This example assumes that the
-data is modified in some undisclosed manner by the ```./modify-data.sh```
+data is modified in some undisclosed manner by the ``./modify-data.sh``
 script which is part of your testdef_repo before the receiver is notified.
-Any packages required by ```modify-data.sh``` need to be added to the install
-deps of sender.yaml. The contents of ```modify-data.sh``` are left as an exercise for the reader.
 
-The LAVA MultiNode API provides ways of querying the network information of devices
+The LAVA :ref:`multinode_api` provides ways of querying the network information of devices
 within the group. In order to offer the data via apache, the sender needs to
 raise a suitable network interface, so it calls ifconfig as a lava test case
 first and then uses the lava-network API call to broadcast network information
@@ -159,33 +201,40 @@
 get-data.sh
 ^^^^^^^^^^^
 
+Always use **set -x** in any wrapper / helper scripts which you expect
+to use in a test run to be able to debug test failures.
+
+Ensure that the scripts are marked as executable in your VCS and
+that the appropriate interpreter is installed in your test image.
+
 ::
 
  #!/bin/sh
  set -e
- DEVICE=`lava-group | grep -m1 -v kvm|cut -f2`
+ set -x
+ DEVICE=`lava-group | grep -m1 receiver|cut -f2`
  SOURCE=`lava-network query $DEVICE ipv4|grep -v LAVA|cut -d: -f2`
  wget -O /tmp/testfile http://${SOURCE}/testfile
 
 
-The ```$DEVICE``` simply matches the first device name in this group
-which contains the string 'kvm' and returns the full name of that device,
-e.g. multinode-kvm02
+The ``$DEVICE`` simply matches the first device name in this group
+which contains the string 'receiver' (which comes from the ``role``
+specified in the JSON) and returns the full name of that device,
+e.g. multinode-kvm02 or staging-beagleblack03
 
 This device name is then passed to lava-network query to get the ipv4
-details of that device within this group. The value of ```$SOURCE```
+details of that device within this group. The value of ``$SOURCE``
 is an IPv4 address of the sender (assuming that your JSON has defined a
-device_type for the sender as a device which would contain
-the 'kvm' string in the name.)
+role for the sender which would contain the 'receiver' string in the name.)
 
-Finally, ```get-data.sh``` does the work of receiving the data from
+Finally, ``get-data.sh`` does the work of receiving the data from
 the sender. The verification of the data is left as an exercise for
 the reader - one simple method would be for the sender to checksum the
-(modified) data and use ```lava-send``` to make that checksum available
-to devices within the group. The receiver can then use ```lava-wait```
+(modified) data and use ``lava-send`` to make that checksum available
+to devices within the group. The receiver can then use ``lava-wait``
 to get that checksum.
 
-Once ```get-data.sh``` returns, the receiver notifies the sender that
+Once ``get-data.sh`` returns, the receiver notifies the sender that
 the transfer is complete, processes the data as it sees fit and cleans up.
 
 Preparing the JSON
@@ -193,7 +242,7 @@
 
 The JSON ties the YAML test definition with the hardware and software to
 run the test definition. The JSON is also where multiple test
-definitions are combined into a single !MultiNode test.
+definitions are combined into a single MultiNode test.
 
 General settings
 ----------------
@@ -289,8 +338,7 @@
 If any action has no role specified, it will be actioned for all roles.
 
 For Use Case One, we have a different YAML file for each role, so
-we have two lava_test_shell commands. (The content happens to be the
-same in this example.)
+we have two lava_test_shell commands.
 
 ::
 
@@ -326,7 +374,8 @@
 ^^^^^^^^^^^^^^
 
 The results for the entire group get aggregated into a single result
-bundle.
+bundle. Ensure that the bundle stream exists on the specified server
+and that you have permission to add to that stream.
 
 ::
 
@@ -334,7 +383,7 @@
         {
             "command": "submit_results_on_host",
             "parameters": {
-                "stream": "/anonymous/instance-manager/",
+                "stream": "/anonymous/use-cases/",
                 "server": "http://validation.linaro.org/RPC2/"
             }
         }
@@ -343,7 +392,7 @@
 Prepare a filter for the results
 ================================
 
-Now decide how your are going to analyse the results of tests using
+Now decide how you are going to analyse the results of tests using
 this definition, using the name of the test definition specified in
 the YAML metadata.
 
@@ -385,7 +434,7 @@
 
 On the website for the instance running the tests, click on Dashboard
 and Filters. If you have permissions, there will be a link entitled
-*Add new filter...*. 
+*Add new filter...*.
 
 The filter name should include most of the data about what this filter
 is intended to do, without whitespace. This name will be preserved through
@@ -406,13 +455,13 @@
 
 Within a test definition, a filter can also select only particular test
 cases. In this Use Case, for example, the filter could choose only the
-```multinode-network```, ```multinode-get-network``` or ```file-sync```
+``multinode-network``, ``multinode-get-network`` or ``file-sync``
 test cases. Continue to add tests and/or test cases - the more tests
 and/or test cases are added to the filter, the fewer results will
 match.
 
 Click the *Preview* button to apply the filter to the current set of
-results **without saving the filter**. 
+results **without saving the filter**.
 
 In the preview, if there are columns with no data or rows with no data
 for specific columns, these will show up as missing data in the filter
@@ -450,7 +499,7 @@
 
 The full version of this use case are available:
 
-http://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=json/kvm-beagleblack-group.json;hb=refs/heads/master
+http://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=json/kvm-beagleblack-group.json;hb=HEAD
 
 Example test results are visible here:
 
@@ -462,3 +511,11 @@
 did not have a bridged configuration, so the internal networking of the kvm meant
 that although the KVM could connect to the beaglebone-black, the beaglebone-black
 could not connect to the kvm.
+
+https://git.linaro.org/gitweb?p=people/neilwilliams/multinode-yaml.git;a=blob_plain;f=json/beagleblack-use-case.json;hb=HEAD
+
+https://staging.validation.linaro.org/dashboard/image-reports/beagleblack-usecase
+
+https://staging.validation.linaro.org/dashboard/streams/anonymous/codehelp/bundles/cf4eb9e0022232e97aaec2737b3cd436cd37ab14/
+
+This example uses two beaglebone-black devices.
 
=== modified file 'doc/usecasetwo.rst'
--- doc/usecasetwo.rst	2013-08-19 10:36:07 +0000
+++ doc/usecasetwo.rst	2013-08-23 10:36:27 +0000
@@ -1,3 +1,5 @@
+.. _use_case_two:
+
 Use Case Two - Setting up the same job on multiple devices
 **********************************************************
 
@@ -7,7 +9,7 @@
 Source Code
 ===========
 
-The test definition itself could be an unchanged singlenode test definition, e.g. 
+The test definition itself could be an unchanged singlenode test definition, e.g.
 
  https://git.linaro.org/gitweb?p=qa/test-definitions.git;a=blob_plain;f=ubuntu/smoke-tests-basic.yaml;hb=refs/heads/master
 
@@ -30,10 +32,9 @@
 Preparing the YAML
 ==================
 
-In this use case, the same YAML file is to be used to test multiple devices.
-Select your YAML file and, if appropriate, edit the name in the metadata.
-
-TBD: MultiNode API
+In the first part of this use case, the same YAML file is to be used to
+test multiple devices. Select your YAML file and, if appropriate, edit
+the name in the metadata.
 
 Preparing the JSON
 ===================
@@ -41,8 +42,8 @@
 The change from a standard single-node JSON file is to expand the device_type
 or device field to a device_group.
 
-The change for multiple devices in MultiNode is within the ```device_group```. To run the test
-multiple devices of the same type, simply increase the ```count``:
+The change for multiple devices in MultiNode is within the ``device_group``. To run the test
+multiple devices of the same type, simply increase the ``count``:
 
 ::
 
@@ -55,15 +56,15 @@
             "tags": [
                 "use-case-two"
             ]
-        } 
+        }
  }
 
-If the rest of the JSON refers to a ```role``` other than the one specified
-in the ```device_group```, those JSON sections are ignored.
+If the rest of the JSON refers to a ``role`` other than the one specified
+in the ``device_group``, those JSON sections are ignored.
 
-If other actions in the JSON do not mention a ```role```, the action will
-occur on all devices in the ```device_group```. So with a single role,
-it only matters that a role exists in the ```device_group```.
+If other actions in the JSON do not mention a ``role``, the action will
+occur on all devices in the ``device_group``. So with a single role,
+it only matters that a role exists in the ``device_group``.
 
 actions
 -------
@@ -136,13 +137,84 @@
 Prepare a filter for the results
 ================================
 
-The filter for this use case uses a ```required attribute``` 
+The filter for this use case uses a ``required attribute``
 of **target.device_type** to only show results for the specified
 devices (to cover reuse of the YAML on other boards later).
 
 It is also possible to add a second filter which matches a specific **target**
 device.
 
+Adding synchronisation
+======================
+
+So far, the multiple devices have been started together but then had no
+further interaction.
+
+The :ref:`multinode_api` supports communication between devices within
+a group and provides synchronisation primitives. The simplest of these
+primitives, :ref:`lava_sync` was used in :ref:`use_case_one` but there are more
+possibilities available.
+
+:ref:`lava_sync` is a special case of a :ref:`lava_send` followed by a
+:ref:`lava_wait_all`.
+
+Sending messages
+----------------
+
+Messages can be sent using :ref:`lava_send` which is a non-blocking call.
+At a later point, another device in the group can collect the message
+using ``lava-wait`` or ``lava-wait-all`` which will block until
+the message is available.
+
+The message can be a simple identifier (e.g. 'download' or 'ready') and
+is visible to all devices in the group.
+
+Key value pairs can also be sent using the API to broadcast particular
+information.
+
+If multiple devices send the same message ID, the data is collated by
+the LAVA Coordinator. Key value pairs sent with any message ID are
+tagged with the device name which sent the key value pairs.
+
+Receiving messages
+------------------
+
+Message reception will block until the message is available.
+
+For :ref:`lava_wait`, the message is deemed available as soon as any device
+in the group has sent a message with the matching ID. If no devices have
+sent such a message, any device asking for ``lava-wait`` on that ID
+will block until a different board uses ``lava-send`` with the expected
+message ID.
+
+For :ref:`lava_wait_all`, the message is only deemed available if **all
+devices in the group** have already sent a message with the expected message
+ID. Therefore, using ``lava-wait-all`` requires a preceding
+``lava-send``.
+
+When using ``lava-wait-all MESSAGEID ROLE``, the message is only deemed
+available if **all devices with the matching role in the group** have
+sent a message with the expected message ID. If the receiving device has
+the specified role, that device must use a ``lava-send`` for the same
+message ID before using ``lava-wait-all MESSAGEID ROLE``.
+
+::
+
+        - lava-test-case multinode-send-network --shell lava-send ready
+        - lava-test-case multinode-get-network --shell lava-wait ready
+
+It is up to the test writer to ensure that when :ref:`lava_wait` is used,
+that the message ID is sufficiently unique that the first use of that
+message ID denotes the correct point in the YAML.
+
+::
+
+        - lava-test-case multinode-send-message --shell lava-send sending source=$(lava-self) role=$(lava-role) hostname=$(hostname -f) kernver=$(uname -r) kernhost=$(uname -n)
+        - lava-test-case multinode-wait-message --shell lava-wait-all sending
+
+This example will wait until all devices in the group have sent the
+message ID ''sending'' (with or without the associated key value pairs).
+
 Summary
 =======