Merge lp:~patrick-crews/drizzle/test-run-docs into lp:drizzle/7.0

[Patrick Crews]

Added sphinx documents for test-run.pl

Includes options + some basic usage examples.

[lp:~patrick-crews/drizzle/test-run-docs]

1894. By Patrick Crews on 2010-11-02

Minor improvements / edits to the test-run documentation

1895. By Patrick Crews on 2010-11-02

Additional minor edits to test-run docs

1896. By Patrick Crews on 2010-11-02

Additional minor edits to test-run docs

1897. By Patrick Crews on 2010-11-02

Added url for reporting bugs in the test-run docs

=== modified file 'docs/include.am'
--- docs/include.am	2010-10-19 00:32:30 +0000
+++ docs/include.am	2010-11-02 18:07:49 +0000
@@ -10,7 +10,8 @@
 	     docs/drizzled_header.html \
 	     ${top_srcdir}/docs/*.rst \
 	     ${top_srcdir}/docs/clients/*.rst \
-	     ${top_srcdir}/docs/libdrizzle/*.rst
+	     ${top_srcdir}/docs/libdrizzle/*.rst \
+             ${top_srcdir}/docs/testing/*.rst
 
 # Makefile for Sphinx documentation
 #

=== modified file 'docs/index.rst'
--- docs/index.rst	2010-10-28 19:50:55 +0000
+++ docs/index.rst	2010-11-02 18:07:49 +0000
@@ -82,6 +82,13 @@
    libdrizzle/api.rst
    libdrizzle/developer.rst
 
+Testing:
+
+.. toctree::
+   :maxdepth: 2
+ 
+   testing/test-run.rst
+
 Indices and tables
 ==================
 

=== added directory 'docs/testing'
=== added file 'docs/testing/test-run.rst'
--- docs/testing/test-run.rst	1970-01-01 00:00:00 +0000
+++ docs/testing/test-run.rst	2010-11-02 18:07:49 +0000
@@ -0,0 +1,472 @@
+**********************************
+test-run.pl - Drizzle testing tool
+**********************************
+
+Synopsis
+========
+
+**./test-run** [ *OPTIONS* ] [ TESTCASE ]
+
+Description
+===========
+
+:program:`test-run.pl` (aka test-run, dtr, mtr) is used to execute tests
+from the Drizzle test suite.  These tests are included with Drizzle 
+distributions and provide a way for users to verify that the system will
+operate according to expectations.
+
+The tests use a diff-based paradigm, meaning that the test runner executes
+a test and then compares the results received with pre-recorded expected 
+results.  In the event of a test failure, the program will provide output
+highlighting the differences found between expected and actual results; this
+can be useful for troubleshooting and in bug reports.
+
+While most users are concerned with ensuring general functionality, the 
+program also allows a user to quickly spin up a server for ad-hoc testing
+and to run the test-suite against an already running Drizzle server.
+
+Running tests
+=========================
+
+There are several different ways to run tests using :program:`test-run.pl`.
+
+It should be noted that unless :option:`--force` is used, the program will
+stop execution upon encountering the first failing test. 
+:option:`--force` is recommended if you are running several tests - it will
+allow you to view all successes and failures in one run.
+
+Running individual tests
+------------------------
+If one only wants to run a few, specific tests, they may do so this way::
+
+    ./test-run [OPTIONS] test1 [test2 ... testN]
+
+Running all tests within a suite
+--------------------------------
+Many of the tests supplied with Drizzle are organized into suites.  
+
+The tests within drizzle/tests/t are considered the 'main' suite.  
+Other suites are located in either drizzle/tests/suite or within the various
+directories in drizzle/plugin.  Tests for a specific plugin should live in 
+the plugin's directory - drizzle/plugin/example_plugin/tests
+
+To run the tests in a specific suite::
+
+    ./test-run [OPTIONS] --suite=SUITENAME
+
+Running specific tests within a suite
+--------------------------------------
+To run a specific set of tests within a suite::
+
+    ./test-run [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
+
+Calling tests using <suitename>.<testname> currently does not work.
+One must specify the test suite via the :option:`--suite` option.
+
+
+Running all available tests
+---------------------------
+Currently, the quickest way to execute all tests in all suites is
+to use 'make test' from the drizzle root.
+
+Otherwise, one should simply name all suites::
+
+    ./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
+
+Interpreting test results
+=========================
+The output of the test runner is quite simple.  Every test should pass.
+In the event of a test failure, please take the time to file a bug here:
+*https://bugs.launchpad.net/drizzle*
+
+During a run, the program will provide the user with:
+  * test name (suite + name)
+  * test status (pass/fail/skipped)
+  * time spent executing each test
+
+At the end of a run, the program will provide the user with a listing of:
+  * how many tests were run
+  * how many tests failed
+  * percentage of passing tests
+  * a listing of failing tests
+  * total time spent executing the tests
+
+Example output::
+
+    <snip>
+    main.snowman                                                 [ pass ]       9
+    main.statement_boundaries                                    [ pass ]      17
+    main.status                                                  [ pass ]      12
+    main.strict                                                  [ pass ]      50
+    main.subselect                                               [ pass ]    6778
+    main.subselect2                                              [ pass ]      51
+    main.subselect3                                              [ fail ]
+    drizzletest: At line 621: query 'select a, (select max(b) from t1) into outfile
+    <snip>
+    --------------------------------------------------------------------------------
+    Stopping All Servers
+    Failed 10/231 tests, 95.67% were successful.
+
+    The log files in var/log may give you some hint
+    of what went wrong.
+    If you want to report this error, go to:
+	http://bugs.launchpad.net/drizzle
+    The servers were restarted 16 times
+    Spent 64.364 of 115 seconds executing testcases
+
+    drizzle-test-run in default mode: *** Failing the test(s): main.exp1 
+    main.func_str main.loaddata main.null main.outfile main.subselect3 
+    main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8
+    
+Additional uses
+===============
+Starting a server for manual testing
+------------------------------------
+
+:program:`test-run.pl` allows a user to get a Drizzle server up and running
+quickly.  This can be useful for fast ad-hoc testing.
+
+To do so call::
+
+    ./test-run --start-and-exit [*OPTIONS*]
+
+This will start a Drizzle server that you can connect to and query
+
+Starting a server against a pre-populated DATADIR
+--------------------------------------------------
+
+Using :option:`--start-dirty` prevents :program:`test-run.pl` from attempting
+to initialize (clean) the datadir.  This can be useful if you want to use
+an already-populated datadir for testing.
+
+Program architecture
+====================
+
+:program:`test-run.pl` uses a simple diff-based mechanism for testing.  
+It will execute the statements contained in a test and compare the results 
+to pre-recorded expected results.  In the event of a test failure, you
+will be presented with a diff::
+
+    main.exp1                                                    [ fail ]
+    --- drizzle/tests/r/exp1.result	2010-11-02 02:10:25.107013998 +0300
+    +++ drizzle/tests/r/exp1.reject	2010-11-02 02:10:32.017013999 +0300
+    @@ -5,4 +5,5 @@
+    a
+    1
+    2
+    +3
+    DROP TABLE t1;
+
+A test case consists of a .test and a .result file.  The .test file includes
+the various statements to be executed for a test.  The .result file lists
+the expected results for a given test file.  These files live in tests/t 
+and tests/r, respectively.  This structure is the same for all test suites.
+
+test-run.pl options
+===================
+
+The :program:`test-run.pl` tool has several available options:
+
+./test-run [ OPTIONS ] [ TESTCASE ]
+
+Options to control what engine/variation to run
+-----------------------------------------------
+
+.. option:: --compress
+   
+   Use the compressed protocol between client and server
+
+.. option:: --bench
+   
+   Run the benchmark suite
+
+.. option:: --small-bench
+
+   Run the benchmarks with --small-tests --small-tables
+
+Options to control directories to use
+-------------------------------------
+
+.. option:: --benchdir=DIR          
+
+   The directory where the benchmark suite is stored
+   (default: ../../mysql-bench)
+  
+.. option:: --tmpdir=DIR
+
+   The directory where temporary files are stored
+   (default: ./var/tmp).
+
+.. option:: --vardir=DIR  
+         
+   The directory where files generated from the test run
+   is stored (default: ./var). Specifying a ramdisk or
+   tmpfs will speed up tests.
+
+.. option:: --mem 
+   
+   Run testsuite in "memory" using tmpfs or ramdisk
+   Attempts to find a suitable location
+   using a builtin list of standard locations
+   for tmpfs (/dev/shm)
+   The option can also be set using environment
+   variable DTR_MEM=[DIR]
+
+Options to control what test suites or cases to run
+---------------------------------------------------
+
+.. option:: --force                 
+   
+   Continue to run the suite after failure
+
+.. option:: --do-test=PREFIX or REGEX
+                        
+   Run test cases which name are prefixed with PREFIX
+   or fulfills REGEX
+
+.. option:: --skip-test=PREFIX or REGEX
+                        
+   Skip test cases which name are prefixed with PREFIX
+   or fulfills REGEX
+
+.. option:: --start-from=PREFIX     
+
+   Run test cases starting from test prefixed with PREFIX
+   suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated
+   list of suite names.
+   The default is: "main,jp"
+
+.. option:: --skip-rpl              
+
+   Skip the replication test cases.
+   combination="ARG1 .. ARG2" Specify a set of "drizzled" arguments for one
+   combination.
+
+.. option:: --skip-combination      
+
+   Skip any combination options and combinations files
+
+.. option:: --repeat-test=n         
+  
+   How many times to repeat each test (default: 1)
+
+Options that specify ports
+--------------------------
+
+.. option:: --master_port=PORT      
+
+   Specify the port number used by the first master
+
+.. option:: --slave_port=PORT      
+
+   Specify the port number used by the first slave
+
+.. option:: --dtr-build-thread=#    
+
+   Specify unique collection of ports. Can also be set by
+   setting the environment variable DTR_BUILD_THREAD.
+
+Options for test case authoring
+-------------------------------
+
+.. option:: --record TESTNAME       
+
+   (Re)genereate the result file for TESTNAME
+
+.. option:: --check-testcases       
+
+   Check testcases for sideeffects
+
+.. option:: --mark-progress         
+
+   Log line number and elapsed time to <testname>.progress
+
+Options that pass on options
+----------------------------
+
+.. option:: --drizzled=ARGS           
+ 
+   Specify additional arguments to "drizzled"
+
+Options to run test on running server
+-------------------------------------
+
+.. option:: --extern                
+
+   Use running server for tests
+
+.. option:: --user=USER             
+
+   User for connection to extern server
+
+Options for debugging the product
+---------------------------------
+
+.. option:: --client-ddd            
+
+   Start drizzletest client in ddd
+
+.. option:: --client-debugger=NAME  
+
+   Start drizzletest in the selected debugger
+
+.. option:: --client-gdb            
+
+   Start drizzletest client in gdb
+
+.. option:: --ddd                   
+
+   Start drizzled in ddd
+
+.. option:: --debug                 
+
+   Dump trace output for all servers and client programs
+
+.. option:: --debugger=NAME         
+
+   Start drizzled in the selected debugger
+
+.. option:: --gdb                   
+
+   Start the drizzled(s) in gdb
+
+.. option:: --manual-debug          
+
+   Let user manually start drizzled in debugger, before running test(s)
+
+.. option:: --manual-gdb            
+
+   Let user manually start drizzled in gdb, before running test(s)
+
+.. option:: --manual-ddd            
+
+   Let user manually start drizzled in ddd, before running test(s)
+
+.. option:: --master-binary=PATH    
+   
+   Specify the master "drizzled" to use
+
+.. option:: --slave-binary=PATH     
+
+   Specify the slave "drizzled" to use
+
+.. option:: --strace-client         
+
+   Create strace output for drizzletest client
+
+.. option:: --max-save-core         
+
+   Limit the number of core files saved (to avoid filling up disks for 
+   heavily crashing server). Defaults to 5, set to 0 for no limit.
+
+Options for coverage, profiling etc
+-----------------------------------
+
+.. option:: --gcov                  
+
+   FIXME
+
+.. option:: --gprof                 
+
+   See online documentation on how to use it.
+
+.. option:: --valgrind              
+
+   Run the *drizzletest* and *drizzled* executables using valgrind with 
+   default options
+
+.. option:: --valgrind-all          
+   
+   Synonym for :option:`--valgrind`
+
+.. option:: --valgrind-drizzleslap  
+
+   Run "drizzleslap" with valgrind.
+
+.. option:: --valgrind-drizzletest  
+
+   Run the *drizzletest* and *drizzle_client_test* executable with valgrind
+
+.. option:: --valgrind-drizzled       
+
+   Run the "drizzled" executable with valgrind
+
+.. option:: --valgrind-options=ARGS 
+
+   Deprecated, use :option:`--valgrind-option`
+
+.. option:: --valgrind-option=ARGS  
+
+   Option to give valgrind, replaces default option(s), 
+   can be specified more then once
+
+.. option:: --valgrind-path=[EXE]   
+
+   Path to the valgrind executable
+
+.. option:: --callgrind             
+
+   Instruct valgrind to use callgrind
+
+.. option:: --massif                
+
+   Instruct valgrind to use massif
+
+Misc options
+------------
+
+.. option:: --comment=STR           
+
+   Write STR to the output
+
+.. option:: --notimer               
+
+   Don't show test case execution time
+
+.. option:: --script-debug          
+
+   Debug this script itself
+
+.. option:: --verbose               
+
+   More verbose output
+
+.. option:: --start-and-exit        
+
+   Only initialize and start the servers, using the
+   startup settings for the specified test case (if any)
+
+.. option:: --start-dirty           
+
+   Only start the servers (without initialization) for
+   the specified test case (if any)
+
+.. option:: --fast                  
+
+   Don't try to clean up from earlier runs
+
+.. option:: --reorder               
+
+   Reorder tests to get fewer server restarts
+
+.. option:: --help                  
+
+   Get this help text
+
+.. option:: --testcase-timeout=MINUTES 
+
+   Max test case run time (default 15)
+
+.. option:: --suite-timeout=MINUTES 
+
+   Max test suite run time (default 180)
+
+.. option:: --warnings | log-warnings 
+
+   Pass --log-warnings to drizzled
+
+.. option:: --sleep=SECONDS         
+
+   Passed to drizzletest, will be used as fixed sleep time
+
+