Merge lp:~linaro-validation/lava-test/connect-training-session-materials into lp:lava-test/0.0

[Zygmunt Krynicki]

This branch adds a few things I wrote for Linaro Connect 12 in Hong Kong, I think they are worth keeping

Unmerged revisions

154. By Zygmunt Krynicki on 2012-05-29

Add INTRO for the lava-test training session at Linaro Connect

=== added file 'INTRO'
--- INTRO	1970-01-01 00:00:00 +0000
+++ INTRO	2012-06-07 08:53:21 +0000
@@ -0,0 +1,289 @@
+This session will cover:
+* What lava-test is.
+* How to install lava-test on a board
+* What tests are available in lava-test, and what they do.
+* How to install and execute tests.
+* How to submit test results to the dashboard.
+* How to find the results in LAVA from a lava-test run.
+* How to wrap existing tests to extend lava-test's capabilities
+* What to keep in mind when designing new tests to allow them to run efficiently in lava-test
+* Where to add tests to lava-test that can be shared.
+* Where to find the source, overview of development rules for new shared tests.
+* how to do private testing
+
+* What lava-test is.
+
+LAVA test is a integration layer between third party test programs / scripts
+and the LAVA infrastructure. Each test wrapper that lava tests can use must
+define how to install, execute and qualify the outcome of a test.
+
+Android also differs in the way it is integrated into LAVA, it is not sharing
+code-base with lava-test as of this time. We plan on integrating both so that
+they share APIs and implementation as much as possible.
+
+* How to install lava-test on a board
+
+There are two ways to install lava-test. You can either use a PPA or install from
+source directly. Since lava-test is written in python is shares the installation
+tools from the python ecosystem. In practice that means installation from
+source is very easy and portable to multiple systems (it works the same on
+debian-based systems, fedora-like systems and windows)
+
+For the purpose of this presentation we will start with a developer or desktop
+image and install from source. We will use the pip tool which is the current
+standard installer for python modules. We'll install pip from the repository to
+get started.
+
+$ sudo apt-get install python-pip
+
+We'll now install lava-test into /usr/local
+
+$ sudo pip install lava-test
+
+You will find that pip has quirks but generally is comparable in spirit with
+apt-get or yum. When we make releases we first publish our source packages
+of lava to python package index (also known as pypi). With pip you should be
+able to get the latest version of lava-test at any time.
+
+NOTE: you will find that many, if not all, python developers use virtualenvs to
+manage their installations. Unless you plan to re-image your board and don't
+care about what's get installed where it's almost always better to use
+virtualenv to manage multiple branches or versions of installed components. For
+simplicity I won't cover that here but I'll be using it in the demo.
+
+* What tests are available in lava-test, and what they do.
+
+LAVA test comes preloaded with a few test wrappers. There are commands
+that allow you to enumerate tests, install or remove tests and so on.
+
+$ lava-test list-tests
+LAVA: Tests built directly into LAVA Test:
+LAVA:  - bluetooth-enablement
+LAVA:  - bootchart
+...
+
+Here lava lists all the tests it knows about, tests are grouped by the provider
+they came from. You can think of providers as wholesale tests merchants. LAVA
+itself has a built-in provider with the tests wrappers it ships with as well as
+two other providers that allow you (end users) to install additional tests in
+two different ways. More advanced users can define a new provider but we won't
+cover that topic here. All you have to know is that lava-test, much like the
+rest of the framework is extensible and loads plug-ins to provide content and
+functionality. 
+
+The LAVA team does not maintain the actual test definitions we ship, which is
+somewhat confusing. We accept patches from people that wish to add their test
+to the default collection but we cannot fix bugs or run them each time they are
+proposed (we only look at the integration side of the code) 
+
+Tests are documented on our documentation page, test maintainers are encouraged
+to provide a description for their test wrappers to help others understand how
+to use them effectively. LAVA has documentation that is maintained in the
+source tree but built for proper presentation by a third party service called
+readthedocs.org
+
+http://lava-test.readthedocs.org/en/latest/tests.html
+
+* How to install and execute tests.
+
+To install a test simply issue the command
+
+$ lava-test install xxx
+
+Where xxx is the name of the test you wish to install. This step will refer
+to the data or code defined in the test wrapper. Typically most tests use the
+default implementation and simply provide a few shell commands to install some
+packages from the archive, download and compile a simple module or something
+similar. Advanced users can freely customize this step to perform any action
+necessary.
+
+LAVA keeps track of the installed tests by creating a directory tree in
+~/.local/share/lava_test/installed-tests/xxx with xxx being replaced by the
+test name. 
+
+* How to submit test results to the dashboard.
+
+This step is a little out of order as we have not covered how to produce test
+results that the dashboard would understand yet. Still it's important to
+discuss how individual developers will submit individual test results vs how
+automated testing (using lava scheduler) submits them. 
+
+Technically both approaches do the same thing. They use the XML-RPC API of the
+lava dashboard and call the put() or put_ex() methods to store the test result
+bundle in a particular bundle stream.
+
+If you are experimenting in your local environment (typically while working on
+a new test wrapper) you will be running a test multiple times, adjusting the
+parser logic or test code, and submitting the result to the dashboard for
+analysis. In that cycle you will be using lava-test to run your test code and
+lava-dashboard-tool to send the test result bundle. The latter tool is simply a
+command line front-end to all of the dashboard XML-RPC functions. It comes with
+built-in documentation (just run it with --help to get started) and is pretty
+easy to use.
+
+Caveat: you will need to create a bundle stream on the dashboard first. You
+can do that from command line by running: 
+
+$ lava-dashboard-tool make-stream /personal/zyga/demo/
+
+You will want to set DASHBOARD_URL _prior_ to running this command, otherwise
+you will have to keep passing the --dashboard-url=... option over and over.
+
+You will _have_ to create an authentication token in the lava web interface
+and associate that token with your machine for the above command to work.
+LAVA uses those tokens to authenticate your requests.
+
+(shows how to navigate to the authentication page and generate a token)
+
+To save your token simply on your machine simply run this command:
+
+$ lava-dashboard-tool auth-add https://[e-mail]/lava-server/RPC2/
+
+It is _essential_ to use https for security (otherwise your token is
+transmitted in plain text). It is also essential to use your username in the
+URL so that lava knows how to match the token to your account. Lastly it is
+essential to use the proper XML-RPC endpoint, including the trailing slash. 
+
+Once the auth-add command finishes successfully you will be able to
+authenticate all your remote requests. You will be able to submit tests results
+to a private bundle streams as well.
+
+So having used the auth-add command I would set my DASHBOARD_URL to 
+export DASHBOARD_URL=https://[e-mail]/lava-server/RPC2/
+(I have something like that in my shell configuration files)
+
+* How to find the results in LAVA from a lava-test run.
+
+Each time you submit a bundle you will get a bundle SHA1 identifier (which is
+just the SHA1 of the entire text of the bundle). You can append that SHA1 to
+a permalink to go directly to that bundle.
+
+Here is an example permalink I pulled from the our production instance:
+
+http://validation.linaro.org/lava-server/dashboard/permalink/bundle/f536c5ce3b0d979be7cdc5934e27e248beb66dff/
+
+You can also navigate to the bundle stream in the web UI and search for the
+bundle ID directly.
+
+* How to wrap existing tests to extend lava-test's capabilities
+
+So this is a pretty simple process but you need to know a few things to be efficient.
+
+First, write down a short description of your test and the requirements it has
+from the test environment. That will be useful to add to the documentation later on.
+
+Write down the installation instructions as a series of shell commands. You will need to
+pass them as an argument to TestInstaller or AndroidTestInstaller. 
+
+Now install your test and write down the run instructions, you will need them
+for the TestRunner or AndroidTestRunner. Run your test and save the output to a file.
+
+Now you should look at the output, figure out what the parts that you want to
+capture are and how to capture them. Typically you will try to write a regular
+expression that describes the text you want to grab. At that point you should
+make a decision: you can either use the built in test parsers (TestParser or
+AndroidTestParser) and tweak the output to be easier to parse or write your
+custom parser in python. Usually starting with line-oriented, record-like
+output is the best as it's trivial to parse. Once this is done it's time to put
+your test together. Don't worry, it's not going to work initially.
+
+You will need to create a 'testobj' that glues your test installer, runner and
+parser together. Now add this test to the list of tests in the project you are
+working on (either built-in test in lava-test/lava-android-test or just another
+test in your custom test project). You may need to regenerate 'egg-info' as
+this is where the data is being stored for python to find. Check that you can
+see your test with lava-test list-tests. If everything is fine just install
+This will allow you to ensure your installation and run instructions are
+correct. Now install your test and run it once.
+
+Now we're ready to work on the regular expression for the default parsers or on
+your full-blown custom parser. We'll be continuously changing the test
+definition (the patterns) and running a command like this:
+
+$ lava-test parse stream stream.2012-05-29T04:54:02Z -o foo.json
+
+Then we'll be looking at foo.json to check if our parser grabbed the data we
+care about. Typically the file is huge because it contains the full log file
+(the entire output of the test) and the environment context (hardware and
+software description). To make it easier to work with we'll re-run the test but pass
+two extra options to it.
+
+$ lava-test run stream -SH
+
+The -S option or --skip-software-context will remove all of the verbose list of
+packages that were installed. The -H option or --skip-hardware-context will
+remove the much shorter but still unneeded at this stage list of devices and
+SoC information.
+
+Make sure you note the result id of the second run. Now re-run parse with that
+result and pass the -A (or --skip-attachments).
+
+$ lava-test parse stream stream.2012-05-29T05:06:59Z -o foo.json -A
+
+Now foo.json is small and tidy and you can instantly see all the data you
+actually care about.
+    
+* What to keep in mind when designing new tests to allow them to run efficiently in lava-test
+
+There are no strong requirements but the things I would think about are:
+
+1) Try not to destroy the machine you are running on, this helps with
+development on a laptop and away from an arm development board. If you need to
+remove everything in /home or do something equally destructive for any reason
+then consider adding a safety argument like --enable-destructive-actions that
+if left unspecified will just abort the process with an appropriate error
+message.
+
+2) Try to have sane output, ease of parsing is paramount. Remember that the
+goal is to capture either test case name (a string) and one of the pass-fail
+keywords or a decimal number. Don't put multiple 'results' on one line. Don't
+use free form test case identifiers, DNS-like names work best (lowercase ASCII
+letters, digits dot and dash).
+
+* Where to add tests to lava-test that can be shared.
+
+You have a few options, the 'correct' choice depends on the nature of those
+tests and the kind of development cycle and maintainership you would like to
+see
+
+1) You can submit them to lava-test or lava-android-test directly. Here they
+should be able to work without special rare hardware or peripherals. Ideally
+they would work on our ubuntu or android images. Typically each merge request
+will take a few days to review and accept. 
+
+2) You can create your own open-source project and keep your tests there. This
+use case makes sense if you plan on providing many tests and want to be free to
+change the code at any time without waiting for the lava team to review your
+changes. I would hope that dedicated testing projects, focused around specific
+topics, would emerge but this has not happened as of this time.
+
+3) If your test is very simple in terms of the wrapper code you can also use a
+declarative test wrapper and simply publish a single .json file somewhere for
+everyone to use.
+
+* Where to find the source, overview of development rules for new shared tests.
+
+It's always worth having a look at the lava documentation on
+http://lava.readthedocs.org/ There are links for lava-test and
+lava-android-test there. In doubt feel free to ask a question on the mailing
+list ([e-mail]) on irc #linaro-lava
+
+Usually it's easier to start with an existing test wrapper, copy it and change
+the things that are relevant to you. Typically a wrapper is very short unless
+it has to change the parser (because the default parser we provide is
+unsuitable for the particular output that the test program generates) 
+
+* how to do private testing
+
+You need to follow three rules:
+
+1) Use https to access your lava instance
+2) Create a token and authenticate your requests with that token
+3) Send your results to private bundle streams (or instruct your jobs to send results to a private bundle stream)
+
+This way you will ensure that:
+
+1) nobody will eavesdrop on your password
+2) nobody will eavesdrop on test results
+3) unauthorized users will not be able to see the details of your test job in lava
+4) unauthorized users will not be able to see the test results in lava