Percona XtraDB Cluster - HA scalable solution for MySQL

Merge lp:~patrick-crews/percona-xtradb-cluster/qp-integrate into lp:~percona-dev/percona-xtradb-cluster/5.5.20

Proposed by Patrick Crews on 2012-02-17
Status: Merged
Approved by: Vadim Tkachenko on 2012-02-17
Approved revision: 3716
Merged at revision: 3718
Proposed branch: lp:~patrick-crews/percona-xtradb-cluster/qp-integrate
Merge into: lp:~percona-dev/percona-xtradb-cluster/5.5.20
Diff against target: 323164 lines
To merge this branch: bzr merge lp:~patrick-crews/percona-xtradb-cluster/qp-integrate
Reviewer Review Type Date Requested Status
Vadim Tkachenko 2012-02-17 Approve on 2012-02-17
Review via email: mp+93648@code.launchpad.net

Description of the Change

Removed dbqp and integrated kewpie (updated / improved dbqp) into the tree.
Cut down on the size to 25MB from 400+.

To execute tests, simply type ./kewpie.py from the kewpie directory (additional options may apply)

To post a comment you must log in.
Vadim Tkachenko (vadim-tk) :
review: Approve

Preview Diff

1=== modified file '.bzrignore'
2--- .bzrignore 2012-02-07 18:26:15 +0000
3+++ .bzrignore 2012-02-17 20:55:33 +0000
4@@ -656,6 +656,8 @@
5 isamchk/*.ds?
6 isamchk/*.vcproj
7 item_xmlfunc.cc
8+kewpie/qp_data/uuid
9+kewpie/workdir
10 lib_debug/*
11 lib_release/*
12 libmysql/*.c
13
14=== removed directory 'dbqp'
15=== removed file 'dbqp/.bzrignore'
16--- dbqp/.bzrignore 2012-02-04 01:22:23 +0000
17+++ dbqp/.bzrignore 1970-01-01 00:00:00 +0000
18@@ -1,3 +0,0 @@
19-dbqp
20-dbqp_data/uuid
21-workdir
22
23=== removed file 'dbqp/dbqp.py'
24--- dbqp/dbqp.py 2012-02-04 01:22:23 +0000
25+++ dbqp/dbqp.py 1970-01-01 00:00:00 +0000
26@@ -1,99 +0,0 @@
27-#! /usr/bin/env python
28-# -*- mode: python; indent-tabs-mode: nil; -*-
29-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
30-#
31-# Copyright (C) 2010 Patrick Crews
32-#
33-# This program is free software; you can redistribute it and/or modify
34-# it under the terms of the GNU General Public License as published by
35-# the Free Software Foundation; either version 2 of the License, or
36-# (at your option) any later version.
37-#
38-# This program is distributed in the hope that it will be useful,
39-# but WITHOUT ANY WARRANTY; without even the implied warranty of
40-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
41-# GNU General Public License for more details.
42-#
43-# You should have received a copy of the GNU General Public License
44-# along with this program; if not, write to the Free Software
45-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
46-
47-
48-""" dbqp.py
49-
50-DataBase Quality Platform - system for executing various
51-testing systems and the helper code
52-
53-Designed to be a modular test-runner. Different testing tools
54-and databases may be plugged into the system via hacking the
55-appropriate modules
56-
57-Currently geared towards Drizzle / will expand to MySQL
58-"""
59-
60-# imports
61-import os
62-import sys
63-
64-import lib.opts.test_run_options as test_run_options
65-from lib.modes.test_mode import handle_mode
66-from lib.server_mgmt.server_management import serverManager
67-from lib.sys_mgmt.system_management import systemManager
68-from lib.test_mgmt.execution_management import executionManager
69-
70-# functions
71-
72-
73-# main
74-variables = test_run_options.variables
75-system_manager = None
76-server_manager = None
77-test_manager = None
78-test_executor = None
79-execution_manager = None
80-
81-try:
82- # Some system-level work is constant regardless
83- # of the test to be run
84- system_manager = systemManager(variables)
85-
86- # Create our server_manager
87- server_manager = serverManager(system_manager, variables)
88-
89- # Get our mode-specific test_manager and test_executor
90- (test_manager,test_executor) = handle_mode(variables, system_manager)
91-
92- # Gather our tests for execution
93- test_manager.gather_tests()
94-
95- # Initialize test execution manager
96- execution_manager = executionManager(server_manager, system_manager
97- , test_manager, test_executor
98- , variables)
99-
100- # Execute our tests!
101- execution_manager.execute_tests()
102-
103-except Exception, e:
104- print Exception, e
105-
106-except KeyboardInterrupt:
107- print "\n\nDetected <Ctrl>+c, shutting down and cleaning up..."
108-
109-finally:
110-# TODO - make a more robust cleanup
111-# At the moment, runaway servers are our biggest concern
112- if server_manager and not variables['startandexit']:
113- if variables['gdb']:
114- server_manager.cleanup_all_servers()
115- else:
116- server_manager.cleanup()
117- if not variables['startandexit']:
118- if test_manager:
119- fail_count = test_manager.has_failing_tests()
120- sys.exit(test_manager.has_failing_tests())
121- else:
122- # return 1 as we likely have a problem if we don't have a
123- # test_manager
124- sys.exit(1)
125-
126
127=== removed directory 'dbqp/dbqp_data'
128=== removed directory 'dbqp/docs'
129=== removed file 'dbqp/docs/dbqp.rst'
130--- dbqp/docs/dbqp.rst 2012-02-04 01:22:23 +0000
131+++ dbqp/docs/dbqp.rst 1970-01-01 00:00:00 +0000
132@@ -1,458 +0,0 @@
133-**********************************
134-dbqp
135-**********************************
136-
137-Synopsis
138-========
139-Drizzle testing tool
140-
141-**./dbqp** [ *OPTIONS* ] [ TESTCASE ]
142-
143-Description
144-===========
145-
146-:program:`dbqp.py` is BETA software. It is intended to provide a standardized
147-platform to facilitate Drizzle testing.
148-
149-The default mode is 'dtr' and is used to execute tests from the Drizzle
150-test suite. These tests are included with Drizzle distributions and
151-provide a way for users to verify that the system will operate according
152-to expectations.
153-
154-The dtr tests use a diff-based paradigm, meaning that the test runner executes
155-a test and then compares the results received with pre-recorded expected
156-results. In the event of a test failure, the program will provide output
157-highlighting the differences found between expected and actual results; this
158-can be useful for troubleshooting and in bug reports.
159-
160-The program is also integrated with the random query generator testing tool
161-and a 'rangden' mode is available - it will execute randgen tests when
162-provided a path to a randgen installation. Tests are organized similar to dtr
163-tests, but are .cnf file based.
164-
165-A 'cleanup' mode is also available as a convenience - it will simply shutdown
166-any servers that may have been started via start-and-exit.
167-
168-While most users are concerned with ensuring general functionality, the
169-program also allows a user to quickly spin up a server for ad-hoc testing
170-and to run the test-suite against an already running Drizzle server.
171-
172-Running tests
173-=========================
174-
175-There are several different ways to run tests using :program:`dbqp.py`.
176-
177-It should be noted that unless :option:`--force` is used, the program will
178-stop execution upon encountering the first failing test.
179-:option:`--force` is recommended if you are running several tests - it will
180-allow you to view all successes and failures in one run.
181-
182-Running individual tests
183-------------------------
184-If one only wants to run a few, specific tests, they may do so this way::
185-
186- ./dbqp.py [OPTIONS] test1 [test2 ... testN]
187-
188-Running all tests within a suite
189---------------------------------
190-Many of the tests supplied with Drizzle are organized into suites.
191-
192-The tests within drizzle/tests/t are considered the 'main' suite.
193-Other suites are located in either drizzle/tests/suite or within the various
194-directories in drizzle/plugin. Tests for a specific plugin should live in
195-the plugin's directory - drizzle/plugin/example_plugin/tests
196-
197-To run the tests in a specific suite::
198-
199- ./dbqp.py [OPTIONS] --suite=SUITENAME
200-
201-Running specific tests within a suite
202---------------------------------------
203-To run a specific set of tests within a suite::
204-
205- ./dbqp.py [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
206-
207-Calling tests using <suitename>.<testname> currently does not work.
208-One must specify the test suite via the :option:`--suite` option.
209-
210-
211-Running all available tests
212----------------------------
213-Currently, the quickest way to execute all tests in all suites is
214-to use 'make test-dbqp' from the drizzle root.
215-
216-Otherwise, one should simply name all suites::
217-
218- ./dbqp.py [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
219-
220-Interpreting test results
221-=========================
222-The output of the test runner is quite simple. Every test should pass.
223-In the event of a test failure, please take the time to file a bug here:
224-*https://bugs.launchpad.net/drizzle*
225-
226-During a run, the program will provide the user with:
227- * test name (suite + name)
228- * test status (pass/fail/skipped)
229- * time spent executing each test
230-
231-At the end of a run, the program will provide the user with a listing of:
232- * how many tests were run
233- * counts and percentages of total exectuted for all test statuses
234- * a listing of failing, skipped, or disabled tests
235- * total time spent executing the tests
236-
237-Example output::
238-
239- <snip>
240- 30 Jan 2011 16:26:31 : main.small_tmp_table [ pass ] 38
241- 30 Jan 2011 16:26:31 : main.snowman [ pass ] 42
242- 30 Jan 2011 16:26:31 : main.statement_boundaries [ pass ] 47
243- 30 Jan 2011 16:26:31 : main.status [ pass ] 51
244- 30 Jan 2011 16:26:31 : main.strict [ pass ] 138
245- 30 Jan 2011 16:26:43 : main.subselect [ fail ] 12361
246- 30 Jan 2011 16:26:43 : --- drizzle/tests/r/subselect.result 2011-01-30 16:23:54.975776148 -0500
247- 30 Jan 2011 16:26:43 : +++ drizzle/tests/r/subselect.reject 2011-01-30 16:26:43.835519303 -0500
248- 30 Jan 2011 16:26:43 : @@ -5,7 +5,7 @@
249- 30 Jan 2011 16:26:43 : 2
250- 30 Jan 2011 16:26:43 : explain extended select (select 2);
251- 30 Jan 2011 16:26:43 : id select_type table type possible_keys key key_len ref rows filtered Extra
252- 30 Jan 2011 16:26:43 : -9 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
253- 30 Jan 2011 16:26:43 : +1 PRIMARY NULL NULL NULL NULL NULL NULL NULL NULL No tables used
254- <snip>
255- 30 Jan 2011 16:30:20 : ================================================================================
256- 30 Jan 2011 16:30:20 INFO: Test execution complete in 314 seconds
257- 30 Jan 2011 16:30:20 INFO: Summary report:
258- 30 Jan 2011 16:30:20 INFO: Executed 552/552 test cases, 100.00 percent
259- 30 Jan 2011 16:30:20 INFO: STATUS: FAIL, 1/552 test cases, 0.18 percent executed
260- 30 Jan 2011 16:30:20 INFO: STATUS: PASS, 551/552 test cases, 99.82 percent executed
261- 30 Jan 2011 16:30:20 INFO: FAIL tests: main.subselect
262- 30 Jan 2011 16:30:20 INFO: Spent 308 / 314 seconds on: TEST(s)
263- 30 Jan 2011 16:30:20 INFO: Test execution complete
264- 30 Jan 2011 16:30:20 INFO: Stopping all running servers...
265-
266-
267-Additional uses
268-===============
269-Starting a server for manual testing
270-------------------------------------
271-
272-:program:`dbqp.py` allows a user to get a Drizzle server up and running
273-quickly. This can be useful for fast ad-hoc testing.
274-
275-To do so call::
276-
277- ./dbqp.py --start-and-exit [*OPTIONS*]
278-
279-This will start a Drizzle server that you can connect to and query
280-
281-Starting a server against a pre-populated DATADIR
282---------------------------------------------------
283-
284-Using :option:`--start-dirty` prevents :program:`dbqp.py` from attempting
285-to initialize (clean) the datadir. This can be useful if you want to use
286-an already-populated datadir for testing.
287-
288-NOTE: This feature is still being tested, use caution with your data!!!
289-
290-Randgen mode / Executing randgen tests
291----------------------------------------
292-
293-Using :option:`--mode` =randgen and :option:`--randgen-path` =/path/to/randgen
294-will cause the randgen tests to execute. This are simple .cnf file-based
295-tests that define various randgen command lines that are useful in testing
296-the server. Test organization is similar to the dtr tests. Tests live in
297-suites, the default suite is 'main' and they all live in
298-drizzle/tests/randgen_tests::
299-
300- ./dbqp.py --mode=randgen --randgen-path=/path/to/randgen
301-
302-A user may specify suites and individual tests to run, just as with dtr-based
303-testing. Test output is the same as well::
304-
305- ./dbqp --mode=randgen --randgen-path=/home/username/repos/randgen
306- Setting --no-secure-file-priv=True for randgen mode...
307- <snip>
308- 23 Feb 2011 11:42:43 INFO: Using testing mode: randgen
309- <snip>
310- 23 Feb 2011 11:44:58 : ================================================================================
311- 23 Feb 2011 11:44:58 : TEST NAME [ RESULT ] TIME (ms)
312- 23 Feb 2011 11:44:58 : ================================================================================
313- 23 Feb 2011 11:44:58 : main.optimizer_subquery [ pass ] 134153
314- 23 Feb 2011 11:45:03 : main.outer_join [ pass ] 5136
315- 23 Feb 2011 11:45:06 : main.simple [ pass ] 2246
316- 23 Feb 2011 11:45:06 : ================================================================================
317- 23 Feb 2011 11:45:06 INFO: Test execution complete in 142 seconds
318- 23 Feb 2011 11:45:06 INFO: Summary report:
319- 23 Feb 2011 11:45:06 INFO: Executed 3/3 test cases, 100.00 percent
320- 23 Feb 2011 11:45:06 INFO: STATUS: PASS, 3/3 test cases, 100.00 percent executed
321- 23 Feb 2011 11:45:06 INFO: Spent 141 / 142 seconds on: TEST(s)
322- 23 Feb 2011 11:45:06 INFO: Test execution complete
323- 23 Feb 2011 11:45:06 INFO: Stopping all running servers...
324-
325-Cleanup mode
326--------------
327-A cleanup mode is provided for user convenience. This simply shuts down
328-any servers whose pid files are detected in the dbqp workdir. It is mainly
329-intended as a quick cleanup for post-testing with :option:`--start-and-exit`::
330-
331- ./dbqp.py --mode=cleanup
332-
333- Setting --start-dirty=True for cleanup mode...
334- 23 Feb 2011 11:35:59 INFO: Using Drizzle source tree:
335- 23 Feb 2011 11:35:59 INFO: basedir: drizzle
336- 23 Feb 2011 11:35:59 INFO: clientbindir: drizzle/client
337- 23 Feb 2011 11:35:59 INFO: testdir: drizzle/tests
338- 23 Feb 2011 11:35:59 INFO: server_version: 2011.02.2188
339- 23 Feb 2011 11:35:59 INFO: server_compile_os: unknown-linux-gnu
340- 23 Feb 2011 11:35:59 INFO: server_platform: x86_64
341- 23 Feb 2011 11:35:59 INFO: server_comment: (Source distribution (dbqp_randgen))
342- 23 Feb 2011 11:35:59 INFO: Using --start-dirty, not attempting to touch directories
343- 23 Feb 2011 11:35:59 INFO: Using default-storage-engine: innodb
344- 23 Feb 2011 11:35:59 INFO: Using testing mode: cleanup
345- 23 Feb 2011 11:35:59 INFO: Killing pid 10484 from drizzle/tests/workdir/testbot0/server0/var/run/server0.pid
346- 23 Feb 2011 11:35:59 INFO: Stopping all running servers...
347-
348-Program architecture
349-====================
350-
351-:program:`dbqp.py`'s 'dtr' mode uses a simple diff-based mechanism for testing.
352-This is the default mode and where the majority of Drizzle testing occurs.
353-It will execute the statements contained in a test and compare the results
354-to pre-recorded expected results. In the event of a test failure, you
355-will be presented with a diff::
356-
357- main.exp1 [ fail ]
358- --- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
359- +++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
360- @@ -5,4 +5,5 @@
361- a
362- 1
363- 2
364- +3
365- DROP TABLE t1;
366-
367-A test case consists of a .test and a .result file. The .test file includes
368-the various statements to be executed for a test. The .result file lists
369-the expected results for a given test file. These files live in tests/t
370-and tests/r, respectively. This structure is the same for all test suites.
371-
372-dbqp.py options
373-===================
374-
375-The :program:`dbqp.py` tool has several available options:
376-
377-./dbqp.py [ OPTIONS ] [ TESTCASE ]
378-
379-
380-Options
381--------
382-
383-.. program:: dbqp.py
384-
385-.. option:: -h, --help
386-
387- show this help message and exit
388-
389-Options for the test-runner itself
390-----------------------------------
391-
392-.. program:: dbqp.py
393-
394-.. option:: --force
395-
396- Set this to continue test execution beyond the first failed test
397-
398-.. option:: --start-and-exit
399-
400- Spin up the server(s) for the first specified test then exit
401- (will leave servers running)
402-
403-.. option:: --verbose
404-
405- Produces extensive output about test-runner state.
406- Distinct from --debug
407-
408-.. option:: --debug
409-
410- Provide internal-level debugging output.
411- Distinct from --verbose
412-
413-.. option:: --mode=MODE
414-
415- Testing mode.
416- We only support dtr...for now >;)
417- [dtr]
418-
419-.. option:: --record
420-
421- Record a testcase result
422- (if the testing mode supports it)
423- [False]
424-
425-.. option:: --fast
426-
427- Don't try to cleanup from earlier runs
428- (currently just a placeholder) [False]
429-
430-.. option:: --randgen-path=RANDGENPATH
431-
432- The path to a randgen installation that can be used to
433- execute randgen-based tests
434-
435-
436-Options for controlling which tests are executed
437-------------------------------------------------
438-
439-.. program:: dbqp.py
440-
441-.. option:: --suite=SUITELIST
442-
443- The name of the suite containing tests we want.
444- Can accept comma-separated list (with no spaces).
445- Additional --suite args are appended to existing list
446- [autosearch]
447-
448-.. option:: --suitepath=SUITEPATHS
449-
450- The path containing the suite(s) you wish to execute.
451- Use on --suitepath for each suite you want to use.
452-
453-.. option:: --do-test=DOTEST
454-
455- input can either be a prefix or a regex.
456- Will only execute tests that match the provided pattern
457-
458-.. option:: --skip-test=SKIPTEST
459-
460- input can either be a prefix or a regex.
461- Will exclude tests that match the provided pattern
462-
463-.. option:: --reorder
464-
465- sort the testcases so that they are executed optimally
466- for the given mode [False]
467-
468-.. option:: --repeat=REPEAT
469-
470- Run each test case the specified number of times. For
471- a given sequence, the first test will be run n times,
472- then the second, etc [1]
473-
474-Options for defining the code that will be under test
475------------------------------------------------------
476-
477-.. program:: dbqp.py
478-
479-.. option:: --basedir=BASEDIR
480-
481- Pass this argument to signal to the test-runner
482- that this is an in-tree test (not required).
483- We automatically set a number of variables
484- relative to the argument (client-bindir,
485- serverdir, testdir) [../]
486-
487-.. option:: --serverdir=SERVERPATH
488-
489- Path to the server executable. [auto-search]
490-
491-.. option:: --client-bindir=CLIENTBINDIR
492-
493- Path to the directory containing client program
494- binaries for use in testing [auto-search]
495-
496-.. option:: --default-storage-engine=DEFAULTENGINE
497-
498- Start drizzled using the specified engine [innodb]
499-
500-Options for defining the testing environment
501---------------------------------------------
502-
503-.. program:: dbqp.py
504-
505-.. option:: --testdir=TESTDIR
506-
507- Path to the test dir, containing additional files for
508- test execution. [pwd]
509-
510-.. option:: --workdir=WORKDIR
511-
512- Path to the directory test-run will use to store
513- generated files and directories.
514- [basedir/tests/dbqp_work]
515-
516-.. option:: --top-srcdir=TOPSRCDIR
517-
518- build option [basedir_default]
519-
520-.. option:: --top-builddir=TOPBUILDDIR
521-
522- build option [basedir_default]
523-
524-.. option:: --no-shm
525-
526- By default, we symlink workdir to a location in shm.
527- Use this flag to not symlink [False]
528-
529-.. option:: --start-dirty
530-
531- Don't try to clean up working directories before test
532- execution [False]
533-
534-.. option:: --no-secure-file-priv
535-
536- Turn off the use of --secure-file-priv=vardir for
537- started servers
538-
539-Options to pass options on to the server
540------------------------------------------
541-
542-.. program:: dbqp.py
543-
544-.. option:: --drizzled=DRIZZLEDOPTIONS
545-
546- Pass additional options to the server. Will be passed
547- to all servers for all tests (mostly for --start-and-
548- exit)
549-
550-
551-Options for defining the tools we use for code analysis (valgrind, gprof, gcov, etc)
552-------------------------------------------------------------------------------------
553-
554-.. program:: dbqp.py
555-
556-.. option:: --valgrind
557-
558- Run drizzletest and drizzled executables using
559- valgrind with default options [False]
560-
561-.. option:: --valgrind-option=VALGRINDARGLIST
562-
563- Pass an option to valgrind (overrides/removes default
564- valgrind options)
565-
566-Options for controlling the use of debuggers with test execution
567-----------------------------------------------------------------
568-
569-.. program:: dbqp.py
570-
571-.. option:: --gdb
572-
573- Start the drizzled server(s) in gdb
574-
575-.. option:: --manual-gdb
576-
577- Allows you to start the drizzled server(s) in gdb
578- manually (in another window, etc
579-
580-Options to call additional utilities such as datagen
581-------------------------------------------------------
582-
583-.. program:: dbqp.py
584-
585-.. option:: --gendata=GENDATAFILE
586-
587- Call the randgen's gendata utility to use the
588- specified configuration file. This will populate the
589- server prior to any test execution
590-
591
592=== removed file 'dbqp/docs/index.rst'
593--- dbqp/docs/index.rst 2012-02-04 01:22:23 +0000
594+++ dbqp/docs/index.rst 1970-01-01 00:00:00 +0000
595@@ -1,33 +0,0 @@
596-.. dbqp documentation master file, created by
597- sphinx-quickstart on Fri Aug 27 08:33:41 2010.
598- You can adapt this file completely to your liking, but it should at least
599- contain the root `toctree` directive.
600-
601-Welcome to dbqp's documentation!
602-===================================
603-
604-dbqp (DataBase Quality Platform) is designed to facilitate testing of MySQL-based database systems. Its aim is to provide a pluggable system that allows one to run a variety of testing tools and to share standard helper code (server allocation and management / test result reporting / etc)
605-
606-Introduction:
607--------------
608-.. toctree::
609- :maxdepth: 2
610-
611-Testing:
612---------
613-.. toctree::
614- :maxdepth: 2
615-
616- dbqp.rst
617- test-run.rst
618- randgen.rst
619- sql-bench.rst
620- sysbench.rst
621- writing_tests.rst
622-
623-Indices and tables
624-==================
625-
626-* :ref:`genindex`
627-* :ref:`search`
628-
629
630=== removed file 'dbqp/docs/randgen.rst'
631--- dbqp/docs/randgen.rst 2012-02-04 01:22:23 +0000
632+++ dbqp/docs/randgen.rst 1970-01-01 00:00:00 +0000
633@@ -1,197 +0,0 @@
634-**********************************
635-randgen (random query generator)
636-**********************************
637-
638-
639-
640-Description
641-===========
642-
643-The randgen aka the random query generator is a database
644-testing tool. It uses a grammar-based stochastic model to represent
645-some desired set of queries (to exercise the optimizer, for example)
646-and generates random queries as allowed by the grammar
647-
648-The primary documentation is here: http://forge.mysql.com/wiki/RandomQueryGenerator
649-
650-This document is intended to help the user set up their environment so that the tool
651-may be used in conjunction with the dbqp.py test-runner. The forge documentation
652-contains more information on the particulars of the tool itself.
653-
654-Requirements
655-============
656-
657-DBD::drizzle
658--------------
659-The DBD::drizzle module is required it can be found here http://launchpad.net/dbd-drizzle/
660-
661-Additional information for installing the module::
662-
663- Prerequisites
664- ----------------
665- * Perl
666- * Drizzle (bzr branch lp:drizzle)
667- * libdrizzle (bzr branch lp:libdrizzle)
668- * C compiler
669-
670- Installation
671- -------------
672- You should only have to run the following:
673-
674- perl Makefile.PL --cflags=-I/usr/local/drizzle/include/ --libs=-"L/usr/local/drizzle/lib -ldrizzle"
675-
676-
677- Depending on where libdrizzle is installed. Also, you'll want to make
678- sure that ldconfig has configured libdrizzle to be in your library path
679-
680-Additional information may be found here: http://forge.mysql.com/wiki/RandomQueryGeneratorQuickStart
681-
682-Installing the randgen
683-=======================
684-
685-The code may be branched from launchpad: bzr branch lp:randgen
686-
687-it also may be downloaded from here http://launchpad.net/randgen/+download
688-
689-That is all there is : )
690-
691-Randgen / dbqp tests
692-====================
693-
694-These tests are simple .cnf files that can define a few basic variables
695-that are needed to execute tests. The most interesting section is test_servers. It is a simple list of python lists
696-Each sub-list contains a string of server options that are needed. Each sub-list represents a server that will be started.
697-Using an empty sub-list will create a server with the default options::
698-
699- [test_info]
700- comment = does NOT actually test the master-slave replication yet, but it will.
701-
702- [test_command]
703- command = ./gentest.pl --gendata=conf/drizzle/drizzle.zz --grammar=conf/drizzle/optimizer_subquery_drizzle.yy --queries=10 --threads=1
704-
705- [test_servers]
706- servers = [[--innodb.replication-log],[--plugin-add=slave --slave.config-file=$MASTER_SERVER_SLAVE_CONFIG]]
707-
708-Running tests
709-=========================
710-
711-There are several different ways to run tests using :doc:`dbqp` 's randgen mode.
712-
713-It should be noted that unless :option:`--force` is used, the program will
714-stop execution upon encountering the first failing test.
715-:option:`--force` is recommended if you are running several tests - it will
716-allow you to view all successes and failures in one run.
717-
718-Running individual tests
719-------------------------
720-If one only wants to run a few, specific tests, they may do so this way::
721-
722- ./dbqp --mode=randgen --randgen-path=/path/to/randgen [OPTIONS] test1 [test2 ... testN]
723-
724-Running all tests within a suite
725---------------------------------
726-Many of the tests supplied with Drizzle are organized into suites.
727-
728-The tests within drizzle/tests/randgen_tests/main are considered the 'main' suite.
729-Other suites are also subdirectories of drizzle/tests/randgen_tests.
730-
731-To run the tests in a specific suite::
732-
733- ./dbqp --mode=randgen --randgen-path=/path/to/randgen [OPTIONS] --suite=SUITENAME
734-
735-Running specific tests within a suite
736---------------------------------------
737-To run a specific set of tests within a suite::
738-
739- ./dbqp --mode=randgen --randgen-path=/path/to/randgen [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
740-
741-Calling tests using <suitename>.<testname> currently does not work.
742-One must specify the test suite via the :option:`--suite` option.
743-
744-
745-Running all available tests
746----------------------------
747-One would currently have to name all suites, but the majority of the working tests live in the main suite
748-Other suites utilize more exotic server combinations and we are currently tweaking them to better integrate with the
749-dbqp system. The slave-plugin suite does currently have a good config file for setting up simple replication setups for testing.
750-To execute several suites' worth of tests::
751-
752- ./dbqp --mode=randgen --randgen-path=/path/to/randgen [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
753-
754-Interpreting test results
755-=========================
756-The output of the test runner is quite simple. Every test should pass.
757-In the event of a test failure, please take the time to file a bug here:
758-*https://bugs.launchpad.net/drizzle*
759-
760-During a run, the program will provide the user with:
761- * test name (suite + name)
762- * test status (pass/fail/skipped)
763- * time spent executing each test
764-
765-At the end of a run, the program will provide the user with a listing of:
766- * how many tests were run
767- * how many tests failed
768- * percentage of passing tests
769- * a listing of failing tests
770- * total time spent executing the tests
771-
772-Example output::
773-
774- 24 Feb 2011 17:27:36 : main.outer_join_portable [ pass ] 7019
775- 24 Feb 2011 17:27:39 : main.repeatable_read [ pass ] 2764
776- 24 Feb 2011 17:28:57 : main.select_stability_validator [ pass ] 77946
777- 24 Feb 2011 17:29:01 : main.subquery [ pass ] 4474
778- 24 Feb 2011 17:30:52 : main.subquery_semijoin [ pass ] 110355
779- 24 Feb 2011 17:31:00 : main.subquery_semijoin_nested [ pass ] 8750
780- 24 Feb 2011 17:31:03 : main.varchar [ pass ] 3048
781- 24 Feb 2011 17:31:03 : ================================================================================
782- 24 Feb 2011 17:31:03 INFO: Test execution complete in 288 seconds
783- 24 Feb 2011 17:31:03 INFO: Summary report:
784- 24 Feb 2011 17:31:03 INFO: Executed 18/18 test cases, 100.00 percent
785- 24 Feb 2011 17:31:03 INFO: STATUS: PASS, 18/18 test cases, 100.00 percent executed
786- 24 Feb 2011 17:31:03 INFO: Spent 287 / 288 seconds on: TEST(s)
787- 24 Feb 2011 17:31:03 INFO: Test execution complete
788- 24 Feb 2011 17:31:03 INFO: Stopping all running servers...
789-
790-
791-Additional uses
792-===============
793-Starting a server for manual testing and (optionally) populating it
794---------------------------------------------------------------------
795-
796-:doc:`dbqp` 's randgen mode allows a user to get a Drizzle server up and running quickly. This can be useful for fast ad-hoc testing.
797-
798-To do so call::
799-
800- ./dbqp --mode=randgen --randgen-path=/path/to/randgen --start-and-exit [*OPTIONS*]
801-
802-This will start a Drizzle server that you can connect to and query
803-
804-With the addition of the --gendata option, a user may utilize the randgen's gendata (table creation and population) tool
805-to populate a test server. In the following example, the test server is now populated by the 8 tables listed below::
806-
807- ./dbqp --mode=randgen --randgen-path=/randgen --start-and-exit --gendata=/randgen/conf/drizzle/drizzle.zz
808- <snip>
809- 24 Feb 2011 17:48:48 INFO: NAME: server0
810- 24 Feb 2011 17:48:48 INFO: MASTER_PORT: 9306
811- 24 Feb 2011 17:48:48 INFO: DRIZZLE_TCP_PORT: 9307
812- 24 Feb 2011 17:48:48 INFO: MC_PORT: 9308
813- 24 Feb 2011 17:48:48 INFO: PBMS_PORT: 9309
814- 24 Feb 2011 17:48:48 INFO: RABBITMQ_NODE_PORT: 9310
815- 24 Feb 2011 17:48:48 INFO: VARDIR: /home/pcrews/bzr/work/dbqp_randgen_updates/tests/workdir/testbot0/server0/var
816- 24 Feb 2011 17:48:48 INFO: STATUS: 1
817- # 2011-02-24T17:48:48 Default schema: test
818- # 2011-02-24T17:48:48 Executor initialized, id GenTest::Executor::Drizzle 2011.02.2198 ()
819- # 2011-02-24T17:48:48 # Creating Drizzle table: test.A; engine: ; rows: 0 .
820- # 2011-02-24T17:48:48 # Creating Drizzle table: test.B; engine: ; rows: 0 .
821- # 2011-02-24T17:48:48 # Creating Drizzle table: test.C; engine: ; rows: 1 .
822- # 2011-02-24T17:48:48 # Creating Drizzle table: test.D; engine: ; rows: 1 .
823- # 2011-02-24T17:48:48 # Creating Drizzle table: test.AA; engine: ; rows: 10 .
824- # 2011-02-24T17:48:48 # Creating Drizzle table: test.BB; engine: ; rows: 10 .
825- # 2011-02-24T17:48:48 # Creating Drizzle table: test.CC; engine: ; rows: 100 .
826- # 2011-02-24T17:48:49 # Creating Drizzle table: test.DD; engine: ; rows: 100 .
827- 24 Feb 2011 17:48:49 INFO: User specified --start-and-exit. dbqp.py exiting and leaving servers running...
828-
829-
830-
831
832=== removed file 'dbqp/docs/sql-bench.rst'
833--- dbqp/docs/sql-bench.rst 2012-02-04 01:22:23 +0000
834+++ dbqp/docs/sql-bench.rst 1970-01-01 00:00:00 +0000
835@@ -1,174 +0,0 @@
836-**********************************
837-sql-bench
838-**********************************
839-
840-
841-
842-Description
843-===========
844-dbqp can take advantage of two testing modes offered by the sql-bench tool.
845-
846-The Drizzle code has a copy of this tool set in the tree and the test-runner offers two modes::
847-
848- sql-bench modes
849- ---------------
850- * sqlbench - runs the entire sql-bench test suite and can take a very long time (~45 minutes)
851- * crashme - runs the crash-me tool and reports failure if any tests should not pass
852-
853-
854-Requirements
855-============
856-DBD::drizzle
857--------------
858-The DBD::drizzle module is required it can be found here http://launchpad.net/dbd-drizzle/
859-
860-Additional information for installing the module::
861-
862- Prerequisites
863- ----------------
864- * Perl
865- * Drizzle (bzr branch lp:drizzle)
866- * libdrizzle (bzr branch lp:libdrizzle)
867- * C compiler
868-
869- Installation
870- -------------
871- You should only have to run the following:
872-
873- perl Makefile.PL --cflags=-I/usr/local/drizzle/include/ --libs=-"L/usr/local/drizzle/lib -ldrizzle"
874-
875-
876- Depending on where libdrizzle is installed. Also, you'll want to make
877- sure that ldconfig has configured libdrizzle to be in your library path
878-
879-
880-sql-bench / dbqp tests
881-=======================
882-
883-Currently, there are only two sql-bench test cases for dbqp. As one might expect, main.all_sqlbench_tests executes::
884-
885- run-all-tests --server=drizzle --dir=$DRIZZLE_TEST_WORKDIR --log --connect-options=port=$MASTER_MYPORT --create-options=ENGINE=innodb --user=root
886-
887-against a Drizzle server. The second test case executes the crashme tool against a running server.
888-
889-Test cases are defined in python .cnf files and live in tests/sqlbench_tests.
890-
891-Running tests
892-=========================
893-
894-NOTE: all_sqlbench_tests can take a significant amount of time to execute (45 minutes or so on a decently provisioned laptop)
895-
896-There are several different ways to run tests using :doc:`dbqp` 's sql-bench mode.
897-
898-It should be noted that unless :option:`--force` is used, the program will
899-stop execution upon encountering the first failing test.
900-:option:`--force` is recommended if you are running several tests - it will
901-allow you to view all successes and failures in one run.
902-
903-At present, sql-bench output in a work in progress. It does report a simple pass/fail, but we are working on alternate ways of viewing / storing the results (and for other testing modes as well)
904-
905-
906-Running all tests within a suite
907---------------------------------
908-At present, there is only one test case per suite for sqlbench and crashme modes - that is all that is needed for these tools.
909-To execute the sql-bench test suite::
910-
911- ./dbqp --mode=sqlbench
912-
913-To execute the crash-me test suite::
914-
915- ./dbqp --mode=crashme
916-
917-Interpreting test results
918-=========================
919-The output of the test runner is quite simple. Every test should pass.
920-In the event of a test failure, please take the time to file a bug here:
921-*https://bugs.launchpad.net/drizzle*
922-
923-During a run, the program will provide the user with:
924- * test name (suite + name)
925- * test status (pass/fail/skipped)
926- * time spent executing each test
927-
928-Example sqlbench output::
929-
930- 20110608-135645 ===============================================================
931- 20110608-135645 TEST NAME [ RESULT ] TIME (ms)
932- 20110608-135645 ===============================================================
933- 20110608-135645 main.all_sqlbench_tests [ pass ] 2732007
934- 20110608-135645 Test finished. You can find the result in:
935- 20110608-135645 drizzle/tests/workdir/RUN-drizzle-Linux_2.6.38_9_generic_x86_64
936- 20110608-135645 Benchmark DBD suite: 2.15
937- 20110608-135645 Date of test: 2011-06-08 13:11:10
938- 20110608-135645 Running tests on: Linux 2.6.38-9-generic x86_64
939- 20110608-135645 Arguments: --connect-options=port=9306 --create-options=ENGINE=innodb
940- 20110608-135645 Comments:
941- 20110608-135645 Limits from:
942- 20110608-135645 Server version: Drizzle 2011.06.19.2325
943- 20110608-135645 Optimization: None
944- 20110608-135645 Hardware:
945- 20110608-135645
946- 20110608-135645 alter-table: Total time: 42 wallclock secs ( 0.06 usr 0.04 sys + 0.00 cusr 0.00 csys = 0.10 CPU)
947- 20110608-135645 ATIS: Total time: 22 wallclock secs ( 4.01 usr 0.26 sys + 0.00 cusr 0.00 csys = 4.27 CPU)
948- 20110608-135645 big-tables: Total time: 24 wallclock secs ( 4.16 usr 0.22 sys + 0.00 cusr 0.00 csys = 4.38 CPU)
949- 20110608-135645 connect: Total time: 31 wallclock secs ( 6.81 usr 4.50 sys + 0.00 cusr 0.00 csys = 11.31 CPU)
950- 20110608-135645 create: Total time: 59 wallclock secs ( 2.93 usr 1.65 sys + 0.00 cusr 0.00 csys = 4.58 CPU)
951- 20110608-135645 insert: Total time: 1962 wallclock secs (270.53 usr 66.35 sys + 0.00 cusr 0.00 csys = 336.88 CPU)
952- 20110608-135645 select: Total time: 560 wallclock secs (23.12 usr 4.62 sys + 0.00 cusr 0.00 csys = 27.74 CPU)
953- 20110608-135645 transactions: Total time: 21 wallclock secs ( 2.43 usr 1.98 sys + 0.00 cusr 0.00 csys = 4.41 CPU)
954- 20110608-135645 wisconsin: Total time: 10 wallclock secs ( 2.11 usr 0.52 sys + 0.00 cusr 0.00 csys = 2.63 CPU)
955- 20110608-135645
956- 20110608-135645 All 9 test executed successfully
957- 20110608-135645
958- 20110608-135645 Totals per operation:
959- 20110608-135645 Operation seconds usr sys cpu tests
960- 20110608-135645 alter_table_add 18.00 0.02 0.00 0.02 100
961- 20110608-135645 alter_table_drop 17.00 0.02 0.01 0.03 91
962- 20110608-135645 connect 2.00 1.02 0.51 1.53 2000
963- <snip>
964- 20110608-135645 update_rollback 3.00 0.26 0.23 0.49 100
965- 20110608-135645 update_with_key 73.00 6.70 5.23 11.93 300000
966- 20110608-135645 update_with_key_prefix 34.00 4.45 2.30 6.75 100000
967- 20110608-135645 wisc_benchmark 2.00 1.49 0.00 1.49 114
968- 20110608-135645 TOTALS 2865.00 310.26 79.94 390.20 2974250
969- 20110608-135645
970- 20110608-135645 ===============================================================
971- 20110608-135645 INFO Test execution complete in 2735 seconds
972- 20110608-135645 INFO Summary report:
973- 20110608-135645 INFO Executed 1/1 test cases, 100.00 percent
974- 20110608-135645 INFO STATUS: PASS, 1/1 test cases, 100.00 percent executed
975- 20110608-135645 INFO Spent 2732 / 2735 seconds on: TEST(s)
976- 20110608-135645 INFO Test execution complete
977- 20110608-135645 INFO Stopping all running servers...
978-
979-Example crashme output::
980-
981- 20110608-152759 ===============================================================
982- 20110608-152759 TEST NAME [ RESULT ] TIME (ms)
983- 20110608-152759 ===============================================================
984- 20110608-152759 main.crashme [ fail ] 155298
985- 20110608-152759 func_extra_to_days=error # Function TO_DAYS
986- 20110608-152759 ###
987- 20110608-152759 ###<select to_days('1996-01-01') from crash_me_d
988- 20110608-152759 ###>2450084
989- 20110608-152759 ###We expected '729024' but got '2450084'
990- 20110608-152759 func_odbc_timestampadd=error # Function TIMESTAMPADD
991- 20110608-152759 ###
992- 20110608-152759 ###<select timestampadd(SQL_TSI_SECOND,1,'1997-01-01 00:00:00')
993- 20110608-152759 ###>1997-01-01 00:00:01.000000
994- 20110608-152759 ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
995- 20110608-152759 ###
996- 20110608-152759 ###<select {fn timestampadd(SQL_TSI_SECOND,1,{ts '1997-01-01 00:00:00'}) }
997- 20110608-152759 ###>1997-01-01 00:00:01.000000
998- 20110608-152759 ###We expected '1997-01-01 00:00:01' but got '1997-01-01 00:00:01.000000'
999- 20110608-152759
1000- 20110608-152759 ERROR Failed test. Use --force to execute beyond the first test failure
1001- 20110608-152759 ===============================================================
1002- 20110608-152759 INFO Test execution complete in 158 seconds
1003- 20110608-152759 INFO Summary report:
1004- 20110608-152759 INFO Executed 1/1 test cases, 100.00 percent
1005- 20110608-152759 INFO STATUS: FAIL, 1/1 test cases, 100.00 percent executed
1006- 20110608-152759 INFO FAIL tests: main.crashme
1007- 20110608-152759 INFO Spent 155 / 158 seconds on: TEST(s)
1008- 20110608-152759 INFO Test execution complete
1009-
1010
1011=== removed file 'dbqp/docs/sysbench.rst'
1012--- dbqp/docs/sysbench.rst 2012-02-04 01:22:23 +0000
1013+++ dbqp/docs/sysbench.rst 1970-01-01 00:00:00 +0000
1014@@ -1,123 +0,0 @@
1015-**********************************
1016-sysbench
1017-**********************************
1018-
1019-
1020-
1021-Description
1022-===========
1023-dbqp's sysbench mode allows a user to run a specific iteration of a sysbench test (eg an oltp readonly run at concurrency = 16)
1024-
1025-
1026-Requirements
1027-============
1028-
1029-The SYSBENCH command requires that the Drizzle library header be installed. Simply build Drizzle and do::
1030-
1031- $> sudo make install
1032-
1033-This will install the right headers.
1034-
1035-The SYSBENCH command also requires installation of the drizzle-sysbench program, which can be installed from source like so::
1036-
1037- $> bzr branch lp:~drizzle-developers/sysbench/trunk drizzle-sysbench
1038- $> cd drizzle-sysbench
1039- $> ./autogen.sh && ./configure && make && sudo make install
1040-
1041-Make sure sysbench is then in your path
1042-
1043-
1044-sysbench / dbqp tests
1045-=====================
1046-
1047-A sysbench test defines a run for a particular concurrency. There are suites for readonly and readwrite.
1048-They are currently broken down this way as an experiment - we are open to other ways of organizing these tests::
1049-
1050- [test_info]
1051- comment = 16 threads
1052-
1053- [test_command]
1054- command = sysbench --max-time=240 --max-requests=0 --test=oltp --db-ps-mode=disable --drizzle-table-engine=innodb --oltp-read-only=on --oltp-table-size=1000000 --drizzle-mysql=on --drizzle-user=root --drizzle-db=test --drizzle-port=$MASTER_MYPORT --drizzle-host=localhost --db-driver=drizzle --num-threads=16
1055-
1056- [test_servers]
1057- servers = [[innodb.buffer-pool-size=256M innodb.log-file-size=64M innodb.log-buffer-size=8M innodb.thread-concurrency=0 innodb.additional-mem-pool-size=16M table-open-cache=4096 table-definition-cache=4096 mysql-protocol.max-connections=2048]]
1058-
1059-Running tests
1060-=========================
1061-
1062-There are several different ways to run tests using :doc:`dbqp` 's sysbench mode.
1063-
1064-It should be noted that unless :option:`--force` is used, the program will
1065-stop execution upon encountering the first failing test.
1066-:option:`--force` is recommended if you are running several tests - it will
1067-allow you to view all successes and failures in one run.
1068-
1069-Running individual tests
1070-------------------------
1071-If one only wants to run a few, specific tests, they may do so this way::
1072-
1073- ./dbqp --mode=sysbench [OPTIONS] test1 [test2 ... testN]
1074-
1075-Running all tests within a suite
1076---------------------------------
1077-Many of the tests supplied with Drizzle are organized into suites.
1078-
1079-The tests within drizzle/tests/randgen_tests/main are considered the 'main' suite.
1080-Other suites are also subdirectories of drizzle/tests/randgen_tests.
1081-
1082-To run the tests in a specific suite::
1083-
1084- ./dbqp --mode=sysbench [OPTIONS] --suite=SUITENAME
1085-
1086-Running specific tests within a suite
1087---------------------------------------
1088-To run a specific set of tests within a suite::
1089-
1090- ./dbqp --mode=sysbench [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
1091-
1092-Calling tests using <suitename>.<testname> currently does not work.
1093-One must specify the test suite via the :option:`--suite` option.
1094-
1095-
1096-Running all available tests
1097----------------------------
1098-One would currently have to name all suites, but the majority of the working tests live in the main suite
1099-Other suites utilize more exotic server combinations and we are currently tweaking them to better integrate with the
1100-dbqp system. The slave-plugin suite does currently have a good config file for setting up simple replication setups for testing.
1101-To execute several suites' worth of tests::
1102-
1103- ./dbqp --mode=sysbench [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
1104-
1105-Interpreting test results
1106-=========================
1107-The output of the test runner is quite simple. Every test should pass.
1108-In the event of a test failure, please take the time to file a bug here:
1109-*https://bugs.launchpad.net/drizzle*
1110-
1111-During a run, the program will provide the user with:
1112- * test name (suite + name)
1113- * test status (pass/fail/skipped)
1114- * time spent executing each test
1115-
1116-Example output::
1117-
1118- 20110601-191706 ===============================================================
1119- 20110601-191706 TEST NAME [ RESULT ] TIME (ms)
1120- 20110601-191706 ===============================================================
1121- 20110601-191706 readonly.concurrency_16 [ pass ] 240019
1122- 20110601-191706 max_req_lat_ms: 21.44
1123- 20110601-191706 rwreqps: 4208.2
1124- 20110601-191706 min_req_lat_ms: 6.31
1125- 20110601-191706 deadlocksps: 0.0
1126- 20110601-191706 tps: 150.29
1127- 20110601-191706 avg_req_lat_ms: 6.65
1128- 20110601-191706 95p_req_lat_ms: 7.02
1129- 20110601-191706 ===============================================================
1130- 20110601-191706 INFO Test execution complete in 275 seconds
1131- 20110601-191706 INFO Summary report:
1132- 20110601-191706 INFO Executed 1/1 test cases, 100.00 percent
1133- 20110601-191706 INFO STATUS: PASS, 1/1 test cases, 100.00 percent executed
1134- 20110601-191706 INFO Spent 240 / 275 seconds on: TEST(s)
1135- 20110601-191706 INFO Test execution complete
1136- 20110601-191706 INFO Stopping all running servers...
1137-
1138
1139=== removed file 'dbqp/docs/test-run.rst'
1140--- dbqp/docs/test-run.rst 2012-02-04 01:22:23 +0000
1141+++ dbqp/docs/test-run.rst 1970-01-01 00:00:00 +0000
1142@@ -1,494 +0,0 @@
1143-.. _test-run-label:
1144-
1145-**********************************
1146-test-run.pl - Drizzle testing tool
1147-**********************************
1148-
1149-Synopsis
1150-========
1151-
1152-**./test-run** [ *OPTIONS* ] [ TESTCASE ]
1153-
1154-Description
1155-===========
1156-
1157-:program:`test-run.pl` (aka test-run, dtr, mtr) is used to execute tests
1158-from the Drizzle test suite. These tests are included with Drizzle
1159-distributions and provide a way for users to verify that the system will
1160-operate according to expectations.
1161-
1162-The tests use a diff-based paradigm, meaning that the test runner executes
1163-a test and then compares the results received with pre-recorded expected
1164-results. In the event of a test failure, the program will provide output
1165-highlighting the differences found between expected and actual results; this
1166-can be useful for troubleshooting and in bug reports.
1167-
1168-While most users are concerned with ensuring general functionality, the
1169-program also allows a user to quickly spin up a server for ad-hoc testing
1170-and to run the test-suite against an already running Drizzle server.
1171-
1172-Running tests
1173-=========================
1174-
1175-There are several different ways to run tests using :program:`test-run.pl`.
1176-
1177-It should be noted that unless :option:`--force` is used, the program will
1178-stop execution upon encountering the first failing test.
1179-:option:`--force` is recommended if you are running several tests - it will
1180-allow you to view all successes and failures in one run.
1181-
1182-Running individual tests
1183-------------------------
1184-If one only wants to run a few, specific tests, they may do so this way::
1185-
1186- ./test-run [OPTIONS] test1 [test2 ... testN]
1187-
1188-Running all tests within a suite
1189---------------------------------
1190-Many of the tests supplied with Drizzle are organized into suites.
1191-
1192-The tests within drizzle/tests/t are considered the 'main' suite.
1193-Other suites are located in either drizzle/tests/suite or within the various
1194-directories in drizzle/plugin. Tests for a specific plugin should live in
1195-the plugin's directory - drizzle/plugin/example_plugin/tests
1196-
1197-To run the tests in a specific suite::
1198-
1199- ./test-run [OPTIONS] --suite=SUITENAME
1200-
1201-Running specific tests within a suite
1202---------------------------------------
1203-To run a specific set of tests within a suite::
1204-
1205- ./test-run [OPTIONS] --suite=SUITENAME TEST1 [TEST2..TESTN]
1206-
1207-Calling tests using <suitename>.<testname> currently does not work.
1208-One must specify the test suite via the :option:`--suite` option.
1209-
1210-
1211-Running all available tests
1212----------------------------
1213-Currently, the quickest way to execute all tests in all suites is
1214-to use 'make test' from the drizzle root.
1215-
1216-Otherwise, one should simply name all suites::
1217-
1218- ./test-run [OPTIONS] --suite=SUITE1, SUITE2, ...SUITEN
1219-
1220-Interpreting test results
1221-=========================
1222-The output of the test runner is quite simple. Every test should pass.
1223-In the event of a test failure, please take the time to file a bug here:
1224-*https://bugs.launchpad.net/drizzle*
1225-
1226-During a run, the program will provide the user with:
1227- * test name (suite + name)
1228- * test status (pass/fail/skipped)
1229- * time spent executing each test
1230-
1231-At the end of a run, the program will provide the user with a listing of:
1232- * how many tests were run
1233- * how many tests failed
1234- * percentage of passing tests
1235- * a listing of failing tests
1236- * total time spent executing the tests
1237-
1238-Example output::
1239-
1240- <snip>
1241- main.snowman [ pass ] 9
1242- main.statement_boundaries [ pass ] 17
1243- main.status [ pass ] 12
1244- main.strict [ pass ] 50
1245- main.subselect [ pass ] 6778
1246- main.subselect2 [ pass ] 51
1247- main.subselect3 [ fail ]
1248- drizzletest: At line 621: query 'select a, (select max(b) from t1) into outfile
1249- <snip>
1250- --------------------------------------------------------------------------------
1251- Stopping All Servers
1252- Failed 10/231 tests, 95.67% were successful.
1253-
1254- The log files in var/log may give you some hint
1255- of what went wrong.
1256- If you want to report this error, go to:
1257- http://bugs.launchpad.net/drizzle
1258- The servers were restarted 16 times
1259- Spent 64.364 of 115 seconds executing testcases
1260-
1261- drizzle-test-run in default mode: *** Failing the test(s): main.exp1
1262- main.func_str main.loaddata main.null main.outfile main.subselect3
1263- main.warnings jp.like_utf8 jp.select_utf8 jp.where_utf8
1264-
1265-Additional uses
1266-===============
1267-Starting a server for manual testing
1268-------------------------------------
1269-
1270-:program:`test-run.pl` allows a user to get a Drizzle server up and running
1271-quickly. This can be useful for fast ad-hoc testing.
1272-
1273-To do so call::
1274-
1275- ./test-run --start-and-exit [*OPTIONS*]
1276-
1277-This will start a Drizzle server that you can connect to and query
1278-
1279-Starting a server against a pre-populated DATADIR
1280---------------------------------------------------
1281-
1282-Using :option:`--start-dirty` prevents :program:`test-run.pl` from attempting
1283-to initialize (clean) the datadir. This can be useful if you want to use
1284-an already-populated datadir for testing.
1285-
1286-Program architecture
1287-====================
1288-
1289-:program:`test-run.pl` uses a simple diff-based mechanism for testing.
1290-It will execute the statements contained in a test and compare the results
1291-to pre-recorded expected results. In the event of a test failure, you
1292-will be presented with a diff::
1293-
1294- main.exp1 [ fail ]
1295- --- drizzle/tests/r/exp1.result 2010-11-02 02:10:25.107013998 +0300
1296- +++ drizzle/tests/r/exp1.reject 2010-11-02 02:10:32.017013999 +0300
1297- @@ -5,4 +5,5 @@
1298- a
1299- 1
1300- 2
1301- +3
1302- DROP TABLE t1;
1303-
1304-A test case consists of a .test and a .result file. The .test file includes
1305-the various statements to be executed for a test. The .result file lists
1306-the expected results for a given test file. These files live in tests/t
1307-and tests/r, respectively. This structure is the same for all test suites.
1308-
1309-test-run.pl options
1310-===================
1311-
1312-The :program:`test-run.pl` tool has several available options:
1313-
1314-./test-run [ OPTIONS ] [ TESTCASE ]
1315-
1316-Options to control what engine/variation to run
1317------------------------------------------------
1318-
1319-.. program:: test-run
1320-
1321-.. option:: --compress
1322-
1323- Use the compressed protocol between client and server
1324-
1325-.. option:: --bench
1326-
1327- Run the benchmark suite
1328-
1329-.. option:: --small-bench
1330-
1331- Run the benchmarks with --small-tests --small-tables
1332-
1333-Options to control directories to use
1334--------------------------------------
1335-
1336-.. program:: test-run
1337-
1338-.. option:: --benchdir=DIR
1339-
1340- The directory where the benchmark suite is stored
1341- (default: ../../mysql-bench)
1342-
1343-.. option:: --tmpdir=DIR
1344-
1345- The directory where temporary files are stored
1346- (default: ./var/tmp).
1347-
1348-.. option:: --vardir=DIR
1349-
1350- The directory where files generated from the test run
1351- is stored (default: ./var). Specifying a ramdisk or
1352- tmpfs will speed up tests.
1353-
1354-.. option:: --mem
1355-
1356- Run testsuite in "memory" using tmpfs or ramdisk
1357- Attempts to find a suitable location
1358- using a builtin list of standard locations
1359- for tmpfs (/dev/shm)
1360- The option can also be set using environment
1361- variable :envvar:`DTR_MEM` =[DIR]
1362-
1363-Options to control what test suites or cases to run
1364----------------------------------------------------
1365-
1366-.. program:: test-run
1367-
1368-.. option:: --force
1369-
1370- Continue to run the suite after failure
1371-
1372-.. option:: --do-test=PREFIX or REGEX
1373-
1374- Run test cases which name are prefixed with PREFIX
1375- or fulfills REGEX
1376-
1377-.. option:: --skip-test=PREFIX or REGEX
1378-
1379- Skip test cases which name are prefixed with PREFIX
1380- or fulfills REGEX
1381-
1382-.. option:: --start-from=PREFIX
1383-
1384- Run test cases starting from test prefixed with PREFIX
1385- suite[s]=NAME1,..,NAMEN Collect tests in suites from the comma separated
1386- list of suite names.
1387- The default is: "main,jp"
1388-
1389-.. option:: --skip-rpl
1390-
1391- Skip the replication test cases.
1392- combination="ARG1 .. ARG2" Specify a set of "drizzled" arguments for one
1393- combination.
1394-
1395-.. option:: --skip-combination
1396-
1397- Skip any combination options and combinations files
1398-
1399-.. option:: --repeat-test=n
1400-
1401- How many times to repeat each test (default: 1)
1402-
1403-Options that specify ports
1404---------------------------
1405-
1406-.. program:: test-run
1407-
1408-.. option:: --master_port=PORT
1409-
1410- Specify the port number used by the first master
1411-
1412-.. option:: --slave_port=PORT
1413-
1414- Specify the port number used by the first slave
1415-
1416-.. option:: --dtr-build-thread=#
1417-
1418- Specify unique collection of ports. Can also be set by
1419- setting the environment variable :envvar:`DTR_BUILD_THREAD`.
1420-
1421-Options for test case authoring
1422--------------------------------
1423-
1424-.. program:: test-run
1425-
1426-.. option:: --record TESTNAME
1427-
1428- (Re)genereate the result file for TESTNAME
1429-
1430-.. option:: --check-testcases
1431-
1432- Check testcases for sideeffects
1433-
1434-.. option:: --mark-progress
1435-
1436- Log line number and elapsed time to <testname>.progress
1437-
1438-Options that pass on options
1439-----------------------------
1440-
1441-.. program:: test-run
1442-
1443-.. option:: --drizzled=ARGS
1444-
1445- Specify additional arguments to "drizzled"
1446-
1447-Options to run test on running server
1448--------------------------------------
1449-
1450-.. program:: test-run
1451-
1452-.. option:: --extern
1453-
1454- Use running server for tests
1455-
1456-.. option:: --user=USER
1457-
1458- User for connection to extern server
1459-
1460-Options for debugging the product
1461----------------------------------
1462-
1463-.. program:: test-run
1464-
1465-.. option:: --client-ddd
1466-
1467- Start drizzletest client in ddd
1468-
1469-.. option:: --client-debugger=NAME
1470-
1471- Start drizzletest in the selected debugger
1472-
1473-.. option:: --client-gdb
1474-
1475- Start drizzletest client in gdb
1476-
1477-.. option:: --ddd
1478-
1479- Start drizzled in ddd
1480-
1481-.. option:: --debug
1482-
1483- Dump trace output for all servers and client programs
1484-
1485-.. option:: --debugger=NAME
1486-
1487- Start drizzled in the selected debugger
1488-
1489-.. option:: --gdb
1490-
1491- Start the drizzled(s) in gdb
1492-
1493-.. option:: --manual-debug
1494-
1495- Let user manually start drizzled in debugger, before running test(s)
1496-
1497-.. option:: --manual-gdb
1498-
1499- Let user manually start drizzled in gdb, before running test(s)
1500-
1501-.. option:: --manual-ddd
1502-
1503- Let user manually start drizzled in ddd, before running test(s)
1504-
1505-.. option:: --master-binary=PATH
1506-
1507- Specify the master "drizzled" to use
1508-
1509-.. option:: --slave-binary=PATH
1510-
1511- Specify the slave "drizzled" to use
1512-
1513-.. option:: --strace-client
1514-
1515- Create strace output for drizzletest client
1516-
1517-.. option:: --max-save-core
1518-
1519- Limit the number of core files saved (to avoid filling up disks for
1520- heavily crashing server). Defaults to 5, set to 0 for no limit.
1521-
1522-Options for coverage, profiling etc
1523------------------------------------
1524-
1525-.. todo::
1526-
1527- .. option:: --gcov
1528-
1529-.. program:: test-run
1530-
1531-.. option:: --gprof
1532-
1533- See online documentation on how to use it.
1534-
1535-.. option:: --valgrind
1536-
1537- Run the *drizzletest* and *drizzled* executables using valgrind with
1538- default options
1539-
1540-.. option:: --valgrind-all
1541-
1542- Synonym for :option:`--valgrind`
1543-
1544-.. option:: --valgrind-drizzleslap
1545-
1546- Run "drizzleslap" with valgrind.
1547-
1548-.. option:: --valgrind-drizzletest
1549-
1550- Run the *drizzletest* and *drizzle_client_test* executable with valgrind
1551-
1552-.. option:: --valgrind-drizzled
1553-
1554- Run the "drizzled" executable with valgrind
1555-
1556-.. option:: --valgrind-options=ARGS
1557-
1558- Deprecated, use :option:`--valgrind-option`
1559-
1560-.. option:: --valgrind-option=ARGS
1561-
1562- Option to give valgrind, replaces default option(s),
1563- can be specified more then once
1564-
1565-.. option:: --valgrind-path=[EXE]
1566-
1567- Path to the valgrind executable
1568-
1569-.. option:: --callgrind
1570-
1571- Instruct valgrind to use callgrind
1572-
1573-.. option:: --massif
1574-
1575- Instruct valgrind to use massif
1576-
1577-Misc options
1578-------------
1579-
1580-.. program:: test-run
1581-
1582-.. option:: --comment=STR
1583-
1584- Write STR to the output
1585-
1586-.. option:: --notimer
1587-
1588- Don't show test case execution time
1589-
1590-.. option:: --script-debug
1591-
1592- Debug this script itself
1593-
1594-.. option:: --verbose
1595-
1596- More verbose output
1597-
1598-.. option:: --start-and-exit
1599-
1600- Only initialize and start the servers, using the
1601- startup settings for the specified test case (if any)
1602-
1603-.. option:: --start-dirty
1604-
1605- Only start the servers (without initialization) for
1606- the specified test case (if any)
1607-
1608-.. option:: --fast
1609-
1610- Don't try to clean up from earlier runs
1611-
1612-.. option:: --reorder
1613-
1614- Reorder tests to get fewer server restarts
1615-
1616-.. option:: --help
1617-
1618- Get this help text
1619-
1620-.. option:: --testcase-timeout=MINUTES
1621-
1622- Max test case run time (default 15)
1623-
1624-.. option:: --suite-timeout=MINUTES
1625-
1626- Max test suite run time (default 180)
1627-
1628-.. option:: --warnings | log-warnings
1629-
1630- Pass --log-warnings to drizzled
1631-
1632-.. option:: --sleep=SECONDS
1633-
1634- Passed to drizzletest, will be used as fixed sleep time
1635-
1636-
1637
1638=== removed file 'dbqp/docs/writing_tests.rst'
1639--- dbqp/docs/writing_tests.rst 2012-02-04 01:22:23 +0000
1640+++ dbqp/docs/writing_tests.rst 1970-01-01 00:00:00 +0000
1641@@ -1,30 +0,0 @@
1642-**********************************
1643-Writing drizzletest test cases
1644-**********************************
1645-
1646-Synopsis
1647-========
1648-
1649-Here, we discuss various topics related to writing test cases, with a focus on features
1650-that allow for more complex testing scenarios. Additional documentation for other testing
1651-tools will come later.
1652-
1653-Using a pre-populated datadir
1654-=============================
1655-The experimental test runner, dbqp allows for starting a server with a pre-populated datadir
1656-for a test case. This is accomplished via the use of a .cnf file (versus a master.opt file)
1657-Over time, this will be the direction for all drizzletest cases.
1658-
1659-The .cnf file shown below tells the test-runner to use the directory drizzle/tests/std_data/backwards_compat_data
1660-as the datadir for the first server. If a test uses multiple servers, the .cnf file can have additional sections ([s1]...[sN])::
1661-
1662- [test_servers]
1663- servers = [[]]
1664-
1665- [s0]
1666- load-datadir=backwards_compat_data
1667-
1668-
1669-All datadirs are expected to be in tests/std_data. If there is need for the ability to use datadirs outside of this location,
1670-it can be explored.
1671-
1672
1673=== removed directory 'dbqp/lib'
1674=== removed file 'dbqp/lib/__init__.py'
1675=== removed directory 'dbqp/lib/modes'
1676=== removed file 'dbqp/lib/modes/__init__.py'
1677=== removed directory 'dbqp/lib/modes/dtr'
1678=== removed file 'dbqp/lib/modes/dtr/__init__.py'
1679=== removed file 'dbqp/lib/modes/dtr/dtr_test_execution.py'
1680--- dbqp/lib/modes/dtr/dtr_test_execution.py 2012-02-04 01:22:23 +0000
1681+++ dbqp/lib/modes/dtr/dtr_test_execution.py 1970-01-01 00:00:00 +0000
1682@@ -1,139 +0,0 @@
1683-#! /usr/bin/env python
1684-# -*- mode: python; indent-tabs-mode: nil; -*-
1685-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
1686-#
1687-# Copyright (C) 2010 Patrick Crews
1688-#
1689-#
1690-# This program is free software; you can redistribute it and/or modify
1691-# it under the terms of the GNU General Public License as published by
1692-# the Free Software Foundation; either version 2 of the License, or
1693-# (at your option) any later version.
1694-#
1695-# This program is distributed in the hope that it will be useful,
1696-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1697-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1698-# GNU General Public License for more details.
1699-#
1700-# You should have received a copy of the GNU General Public License
1701-# along with this program; if not, write to the Free Software
1702-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
1703-
1704-""" dtr_test_execution:
1705- code related to the execution of dtr test cases
1706-
1707- We are provided access to a testManager with
1708- dtr-specific testCases. We contact teh executionManager
1709- to produce the system and server configurations we need
1710- to execute a test.
1711-
1712-"""
1713-
1714-# imports
1715-import os
1716-import sys
1717-import subprocess
1718-import commands
1719-
1720-import lib.test_mgmt.test_execution as test_execution
1721-
1722-class testExecutor(test_execution.testExecutor):
1723- """ dtr-specific testExecutor
1724- We currently execute by sending test-case
1725- data to client/drizzletest...for now
1726-
1727- """
1728-
1729- def execute_testCase (self):
1730- """ Execute a dtr testCase via calls to drizzletest (boo)
1731- Eventually, we will replace drizzletest with pythonic
1732- goodness, but we have these classes stored here for the moment
1733-
1734- """
1735- test_execution.testExecutor.execute_testCase(self)
1736- self.status = 0
1737-
1738- # generate command line
1739- drizzletest_cmd = self.generate_drizzletest_call()
1740-
1741- # call drizzletest
1742- self.execute_drizzletest(drizzletest_cmd)
1743-
1744- # analyze results
1745- self.current_test_status = self.process_drizzletest_output()
1746- self.set_server_status(self.current_test_status)
1747-
1748-
1749- def generate_drizzletest_call(self):
1750- """ Produce the command line we use to call drizzletest
1751- We have a healthy number of values, so we put this in a
1752- nice function
1753-
1754- """
1755-
1756- drizzletest_arguments = [ '--no-defaults'
1757- , '--silent'
1758- , '--tmpdir=%s' %(self.master_server.tmpdir)
1759- , '--logdir=%s' %(self.master_server.logdir)
1760- , '--port=%d' %(self.master_server.master_port)
1761- , '--database=test'
1762- , '--user=root'
1763- , '--password='
1764- #, '--testdir=%s' %(self.test_manager.testdir)
1765- , '--test-file=%s' %(self.current_testcase.testpath)
1766- , '--tail-lines=20'
1767- , '--timer-file=%s' %(self.master_server.timer_file)
1768- , '--result-file=%s' %(self.current_testcase.resultpath)
1769- ]
1770- if self.record_flag:
1771- # We want to record a new result
1772- drizzletest_arguments.append('--record')
1773- drizzletest_cmd = "%s %s %s" %( self.cmd_prefix
1774- , self.master_server.code_tree.drizzletest
1775- , " ".join(drizzletest_arguments))
1776- return drizzletest_cmd
1777-
1778- def execute_drizzletest(self, drizzletest_cmd):
1779- """ Execute the commandline and return the result.
1780- We use subprocess as we can pass os.environ dicts and whatnot
1781-
1782- """
1783- testcase_name = self.current_testcase.fullname
1784- self.time_manager.start(testcase_name,'test')
1785- #retcode, output = self.system_manager.execute_cmd( drizzletest_cmd
1786- # , must_pass = 0 )
1787- drizzletest_outfile = os.path.join(self.logdir,'drizzletest.out')
1788- drizzletest_output = open(drizzletest_outfile,'w')
1789- drizzletest_subproc = subprocess.Popen( drizzletest_cmd
1790- , shell=True
1791- , cwd=self.system_manager.testdir
1792- , env=self.working_environment
1793- , stdout = drizzletest_output
1794- , stderr = subprocess.STDOUT
1795- )
1796- drizzletest_subproc.wait()
1797- retcode = drizzletest_subproc.returncode
1798- execution_time = int(self.time_manager.stop(testcase_name)*1000) # millisec
1799-
1800- drizzletest_output.close()
1801- drizzletest_file = open(drizzletest_outfile,'r')
1802- output = ''.join(drizzletest_file.readlines())
1803- drizzletest_file.close()
1804-
1805- self.logging.debug("drizzletest_retcode: %d" %(retcode))
1806- self.current_test_retcode = retcode
1807- self.current_test_output = output
1808- self.current_test_exec_time = execution_time
1809-
1810- def process_drizzletest_output(self):
1811- """ Drizzletest has run, we now check out what we have """
1812- retcode = self.current_test_retcode
1813- if retcode == 0:
1814- return 'pass'
1815- elif retcode == 62 or retcode == 15872:
1816- return 'skipped'
1817- elif retcode == 63 or retcode == 1:
1818- return 'fail'
1819- else:
1820- return 'fail'
1821-
1822
1823=== removed file 'dbqp/lib/modes/dtr/dtr_test_management.py'
1824--- dbqp/lib/modes/dtr/dtr_test_management.py 2012-02-04 01:22:23 +0000
1825+++ dbqp/lib/modes/dtr/dtr_test_management.py 1970-01-01 00:00:00 +0000
1826@@ -1,495 +0,0 @@
1827-#! /usr/bin/env python
1828-# -*- mode: python; indent-tabs-mode: nil; -*-
1829-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
1830-#
1831-# Copyright (C) 2010 Patrick Crews
1832-#
1833-## This program is free software; you can redistribute it and/or modify
1834-# it under the terms of the GNU General Public License as published by
1835-# the Free Software Foundation; either version 2 of the License, or
1836-# (at your option) any later version.
1837-#
1838-# This program is distributed in the hope that it will be useful,
1839-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1840-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1841-# GNU General Public License for more details.
1842-#
1843-# You should have received a copy of the GNU General Public License
1844-# along with this program; if not, write to the Free Software
1845-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
1846-
1847-""" dtr_test_management:
1848- code related to the gathering / analysis / management of
1849- the test cases
1850- ie - collecting the list of tests in each suite, then
1851- gathering additional, relevant information for the test-runner's dtr
1852- mode. (traditional diff-based testing)
1853-
1854-"""
1855-
1856-# imports
1857-import os
1858-import re
1859-import sys
1860-from ConfigParser import RawConfigParser
1861-
1862-import lib.test_mgmt.test_management as test_management
1863-
1864-
1865-
1866-class testCase:
1867- """Holds info on a per .test file basis
1868- Attributes contain information necessary to execute / validate
1869- the test file when it is executed.
1870-
1871- """
1872- def __init__(self, system_manager, test_case=None, test_name=None, suite_name=None
1873- , suite_path=None, test_server_options=[], test_path=None, result_path=None
1874- , comment=None, master_sh=None, cnf_path=None
1875- , disable=0, innodb_test=1
1876- , need_debug=0, debug=0):
1877- self.system_manager = system_manager
1878- self.logging = self.system_manager.logging
1879- self.skip_keys = ['system_manager']
1880- self.testcase = test_case
1881- self.testname = test_name
1882- self.suitename = suite_name
1883- self.suitepath = suite_path
1884- self.fullname = "%s.%s" %(suite_name, test_name)
1885- self.testpath = test_path
1886- self.resultpath = result_path
1887-
1888- self.skip_flag = 0
1889- self.timezone = "GMT-3"
1890- self.component_id = "drizzled"
1891- self.slave_count = 0
1892- self.master_count = 1
1893- self.server_options = test_server_options
1894- # We will populate this in a better fashion later on
1895- # as we allow .cnf files, we need to do a bit of
1896- # messing about to make this work right
1897- if self.server_options == [] or type(self.server_options[0]) is not list:
1898- self.server_requirements=[self.server_options]
1899- else:
1900- self.server_requirements = self.server_options
1901- self.server_options= self.server_options[0][0]
1902- self.comment = comment
1903- self.master_sh = master_sh
1904- self.cnf_path = cnf_path
1905- self.disable = disable
1906- self.innodb_test = innodb_test
1907- self.need_debug = need_debug
1908-
1909- self.system_manager.logging.debug_class(self)
1910-
1911- def should_run(self):
1912- if self.skip_flag or self.disable:
1913- return 0
1914- else:
1915- return 1
1916-
1917-
1918-
1919-
1920-
1921-class testManager(test_management.testManager):
1922- """Deals with scanning test directories, gathering test cases, and
1923- collecting per-test information (opt files, etc) for use by the
1924- test-runner
1925-
1926- """
1927-
1928- def process_suite(self,suite_dir):
1929- """Process a test suite.
1930- This includes searching for tests in test_list and only
1931- working with the named tests (all tests in suite is the default)
1932- Further processing includes reading the disabled.def file
1933- to know which tests to skip, processing the suite.opt file,
1934- and processing the individual test cases for data relevant
1935- to the rest of the test-runner
1936-
1937- """
1938-
1939- self.system_manager.logging.verbose("Processing suite: %s" %(suite_dir))
1940-
1941- # Generate our test and result files:
1942- testdir = os.path.join(suite_dir, 't')
1943- resultdir = os.path.join(suite_dir, 'r')
1944-
1945- # Do basic checks to make sure this is worth further work
1946- self.check_suite(suite_dir, testdir, resultdir)
1947-
1948- # Get suite-level options
1949- suite_options = []
1950- cnf_path, suite_options = self.process_suite_options(suite_dir)
1951-
1952- # Get the 'name' of the suite. This can require some processing
1953- # But the name is useful for reporting and whatnot
1954- suite_name = self.get_suite_name(suite_dir)
1955-
1956- # Get the contents of the testdir and filter it accordingly
1957- # This applies do-test / skip-test filters and any specific
1958- # test case names
1959- testlist = self.testlist_filter(os.listdir(testdir))
1960- # sort our list if no individual tests were specified
1961- if not self.desired_tests:
1962- testlist.sort()
1963-
1964- # gather deeper information on tests we are interested in
1965- if testlist:
1966- # We have tests we want to process, we gather additional information
1967- # that is useful at the suite level. This includes disabled tests
1968- # Gather disabled test list.
1969- # This is used in process_test_file()
1970- disabled_tests = {}
1971- disabled_tests = self.process_disabled_test_file(testdir)
1972- for test_case in testlist:
1973- self.add_test(self.process_test_file(suite_dir,
1974- suite_name, cnf_path, suite_options
1975- , disabled_tests, testdir
1976- , resultdir, test_case))
1977-
1978- def process_test_file(self, suite_dir, suite_name, suite_cnf_path, suite_options
1979- , disabled_tests, testdir
1980- , resultdir, test_case):
1981- """ We generate / find / store all the relevant information per-test.
1982- This information will be used when actually executing the test
1983- We store the data in a testCase object
1984-
1985- """
1986-
1987- test_server_options = suite_options
1988- test_name = test_case.replace('.test','')
1989- self.system_manager.logging.verbose("Processing test: %s.%s" %(suite_name,test_name))
1990-
1991-
1992- # Fix this , create a testCase with gather_test_data() passed
1993- # as the argument
1994- # Ensure we pass everything we need and use it all
1995- ( test_path
1996- , result_file_name
1997- , result_path
1998- , comment
1999- , master_sh
2000- , cnf_path
2001- , test_server_options
2002- , disable
2003- , innodb_test
2004- , need_debug) = self.gather_test_data(test_case, test_name,
2005- suite_name, test_server_options,testdir,
2006- resultdir, disabled_tests)
2007- if suite_cnf_path and not cnf_path:
2008- cnf_path=suite_cnf_path
2009- test_case = testCase(self.system_manager, test_case, test_name, suite_name,
2010- suite_dir, test_server_options,test_path, result_path,
2011- master_sh=master_sh, cnf_path=cnf_path, debug=self.debug)
2012- return test_case
2013-
2014-
2015-########################################################################
2016-# utility functions
2017-#
2018-# Stuff that helps us out and simplifies our main functions
2019-# But isn't that important unless you need to dig deep
2020-########################################################################
2021-
2022- def gather_test_data(self, test_case, test_name, suite_name,
2023- test_server_options, testdir, resultdir, disabled_tests):
2024- """ We gather all of the data needed to produce a testCase for
2025- a given test
2026-
2027- """
2028-
2029- test_path = os.path.join(testdir,test_case)
2030- result_file_name = test_name+'.result'
2031- result_path = self.find_result_path(resultdir, result_file_name)
2032- comment = None
2033- master_sh_path = test_path.replace('.test','-master.sh')
2034- if os.path.exists(master_sh_path):
2035- master_sh = master_sh_path
2036- else:
2037- master_sh = None
2038- master_opt_path = test_path.replace('.test', '-master.opt')
2039- config_file_path = test_path.replace('.test', '.cnf')
2040- # NOTE: this currently makes suite level server options additive
2041- # to file-level .opt files...not sure if this is the best.
2042- test_server_options = test_server_options + self.process_opt_file(
2043- master_opt_path)
2044- # deal with .cnf files (which supercede master.opt stuff)
2045- cnf_options = []
2046- cnf_flag, returned_options = self.process_cnf_file(config_file_path)
2047- cnf_options += returned_options
2048- if cnf_flag: # we found a proper file and need to override
2049- found_options = cnf_options
2050- else:
2051- config_file_path = None
2052- (disable, comment) = self.check_if_disabled(disabled_tests, test_name)
2053- innodb_test = 0
2054- need_debug = 0
2055- return (test_path, result_file_name, result_path, comment, master_sh,
2056- config_file_path, test_server_options, disable, innodb_test, need_debug)
2057-
2058- def check_suite(self, suite_dir, testdir, resultdir):
2059- """Handle basic checks of the suite:
2060- does the suite exist?
2061- is there a /t dir?
2062- is there a /r dir?
2063- Error and die if not
2064-
2065- """
2066- # We expect suite to be a path name, no fuzzy searching
2067- if not os.path.exists(suite_dir):
2068- self.system_manager.logging.error("Suite: %s does not exist" %(suite_dir))
2069- sys.exit(1)
2070-
2071- # Ensure our test and result directories are present
2072- if not os.path.exists(testdir):
2073- self.system_manager.logging.error("Suite: %s does not have a 't' directory (expected location for test files)" %(suite_dir))
2074- sys.exit(1)
2075- if not os.path.exists(resultdir):
2076- self.system_manager.logging.error("Suite: %s does not have an 'r' directory (expected location for result files)" %(suite_dir))
2077- sys.exit(1)
2078-
2079- def get_suite_name(self, suite_dir):
2080- """ Get the 'name' of the suite
2081- This can either be the path basename or one directory up, as
2082- in the case of files in the drizzle/plugins directory
2083-
2084- """
2085-
2086- # We trim any trailing path delimiters as python returns
2087- # '' for basedir if the path ends that way
2088- # BEGIN horrible hack to accomodate bad location of main suite : /
2089- if suite_dir == self.testdir:
2090- return 'main'
2091- # END horrible hack : /
2092- if suite_dir.endswith('/'):
2093- suite_dir=suite_dir[:-1]
2094- suite_dir_root,suite_dir_basename = os.path.split(suite_dir)
2095- if suite_dir_basename == 'tests' or suite_dir_basename == 'drizzle-tests':
2096- suite_name = os.path.basename(suite_dir_root)
2097- else:
2098- suite_name = suite_dir_basename
2099- self.system_manager.logging.debug("Suite_name: %s" %(suite_name))
2100- return suite_name
2101-
2102- def process_suite_options(self, suite_dir):
2103- """ Process the suite.opt and master.opt files
2104- that reside at the suite-level if they exist.
2105- Return a list of the options found
2106-
2107- We also process .cnf files - this
2108- is currently dbqp-only and is the proper
2109- way to do things :P
2110-
2111- """
2112- found_options = []
2113- opt_files = ['t/master.opt','t/suite.opt']
2114- for opt_file in opt_files:
2115- found_options = found_options + self.process_opt_file(os.path.join(suite_dir,opt_file))
2116- # We also process the suite-level .cnf file(s). We override
2117- # a master.opt file if we have a .cnf file. There is no reason they
2118- # should ever be used in conjunction and I am biased towards .cnf ; )
2119- cnf_files = ['t/master.cnf']
2120- cnf_options = []
2121- for cnf_file in cnf_files:
2122- config_file_path = os.path.join(suite_dir,cnf_file)
2123- cnf_flag, returned_options = self.process_cnf_file(config_file_path)
2124- cnf_options += returned_options
2125- if cnf_flag: # we found a proper file and need to override
2126- found_options = cnf_options
2127- else:
2128- config_file_path = None
2129- return config_file_path, found_options
2130-
2131- def process_disabled_test_file(self, testdir):
2132- """ Checks and processes the suite's disabled.def
2133- file. This file must reside in the suite/t directory
2134- It must be in the format:
2135- test-name : comment (eg BugNNNN - bug on hold, test disabled)
2136- In reality a test should *never* be disabled. EVER.
2137- However, we keep this as a bit of utility
2138-
2139- """
2140- disabled_tests = {}
2141- disabled_def_path = os.path.join(testdir,'disabled.def')
2142- if not os.path.exists(disabled_def_path):
2143- return disabled_tests
2144-
2145- try:
2146- disabled_test_file = open(disabled_def_path,'r')
2147- except IOError, e:
2148- self.system_manager.logging.error("Problem opening disabled.def file: %s" %(disabled_def_path))
2149- sys.exit(1)
2150-
2151- self.system_manager.logging.debug("Processing disabled.def file: %s" %(disabled_def_path))
2152- disabled_bug_pattern = re.compile("[\S]+[\s]+:[\s]+[\S]")
2153-
2154- for line in disabled_test_file:
2155- line = line.strip()
2156- if not line.startswith('#'): # comment
2157- if re.match(disabled_test_pattern,line):
2158- self.system_manager.logging.debug("found disabled test - %s" %(line))
2159- test_name, test_comment = line.split(':')
2160- disabled_tests[test_name.strip()]=test_comment.strip()
2161-
2162- disabled_test_file.close()
2163- return disabled_tests
2164-
2165-
2166- def process_opt_file(self, opt_file_path):
2167- """ Process a test-run '.opt' file.
2168- These files contain test and suite-specific server options
2169- (ie what options the server needs to use for the test)
2170-
2171- Returns a list of the options (we don't really validate...yet)
2172-
2173- NOTE: test-run.pl allows for server *and* system options
2174- (eg timezone, slave_count, etc) in opt files. We don't.
2175- None of our tests use this and we should probably avoid it
2176- We can introduce server and system .opt files or even better
2177- would be to use config files as we do with drizzle-automation
2178- This would allow us to specify options for several different
2179- things in a single file, but in a clean and standardized manner
2180-
2181- """
2182- found_options = []
2183- if not os.path.exists(opt_file_path):
2184- return found_options
2185-
2186- try:
2187- opt_file = open(opt_file_path,'r')
2188- except IOError, e:
2189- self.system_manager.logging.error("Problem opening option file: %s" %(opt_file_path))
2190- sys.exit(1)
2191-
2192- self.system_manager.logging.debug("Processing opt file: %s" %(opt_file_path))
2193- for line in opt_file:
2194- options = line.split('--')
2195- if options:
2196- for option in options:
2197- if option:
2198- if 'restart' in option or '#' in option:
2199- option = 'restart'
2200- found_options.append('--%s' %(option.strip()))
2201- opt_file.close()
2202- return found_options
2203-
2204- def process_cnf_file(self, cnf_file_path):
2205- """ We extract meaningful information from a .cnf file
2206- if it exists. Currently limited to server allocation
2207- needs
2208-
2209- """
2210-
2211- server_requirements = []
2212- cnf_flag = 0
2213- if os.path.exists(cnf_file_path):
2214- cnf_flag = 1
2215- config_reader = RawConfigParser()
2216- config_reader.read(cnf_file_path)
2217- server_requirements = self.process_server_reqs(config_reader.get('test_servers','servers'))
2218- return ( cnf_flag, server_requirements )
2219-
2220- def process_server_reqs(self,data_string):
2221- """ We read in the list of lists as a string, so we need to
2222- handle this / break it down into proper chunks
2223-
2224- """
2225- server_reqs = []
2226- # We expect to see a list of lists and throw away the
2227- # enclosing brackets
2228- option_sets = data_string[1:-1].strip().split(',')
2229- for option_set in option_sets:
2230- server_reqs.append([option_set[1:-1].strip()])
2231- return server_reqs
2232-
2233- def testlist_filter(self, testlist):
2234- """ Filter our list of testdir contents based on several
2235- criteria. This looks for user-specified test-cases
2236- and applies the do-test and skip-test filters
2237-
2238- Returns the list of tests that we want to execute
2239- for further processing
2240-
2241- """
2242-
2243- # We want only .test files
2244- # Possible TODO: allow alternate test extensions
2245- testlist = [test_file for test_file in testlist if test_file.endswith('.test')]
2246-
2247- # Search for specific test names
2248- if self.desired_tests: # We have specific, named tests we want from the suite(s)
2249- tests_to_use = []
2250- for test in self.desired_tests:
2251- if test.endswith('.test'):
2252- pass
2253- else:
2254- test = test+'.test'
2255- if test in testlist:
2256- tests_to_use.append(test)
2257- testlist = tests_to_use
2258-
2259- # TODO: Allow for regex?
2260- # Apply do-test filter
2261- if self.dotest:
2262- testlist = [test_file for test_file in testlist if test_file.startswith(self.dotest)]
2263- # Apply skip-test filter
2264- if self.skiptest:
2265- testlist = [test_file for test_file in testlist if not test_file.startswith(self.skiptest)]
2266- return testlist
2267-
2268- def find_result_path(self, result_dir, result_file_name):
2269- """ This is copied from test-run.pl dtr_cases.pl
2270- If we have an engine option passed in and the
2271- path resultdir/engine/testname.result exists, that is
2272- our .result file
2273-
2274- Need to check if we really need this - maybe PBXT?
2275-
2276- """
2277- result_path = os.path.join(result_dir,result_file_name)
2278- if self.default_engine:
2279- candidate_path = os.path.join(result_dir, self.default_engine,
2280- result_file_name)
2281- if os.path.exists(candidate_path):
2282- result_path = candidate_path
2283- return result_path
2284-
2285- def check_if_disabled(self, disabled_tests, test_name):
2286- """ Scan the list of disabled tests if it exists to see
2287- if the test is disabled.
2288-
2289- """
2290-
2291- if disabled_tests:
2292- if test_name in disabled_tests:
2293- self.system_manager.logging.debug("%s says - I'm disabled" %(test_name))
2294- return (1, disabled_tests[test_name])
2295- return (0,None)
2296-
2297- def sort_testcases(self):
2298- """ We sort our testcases according to the server_options they have
2299- For each testcase, we sort the list of options, so if a test has
2300- --plugin-add=csv --abracadabra, we would get
2301- --abracadabra --plugin-add=csv
2302-
2303- This results in tests that have similar options being run in order
2304- this minimizes server restarts which can be costly
2305-
2306- """
2307- test_management.testManager.sort_testcases(self)
2308- organizer = {}
2309- ordered_list = []
2310- for testcase in self.test_list:
2311- key = " ".join(sorted(testcase.server_options))
2312- if key in organizer:
2313- organizer[key].append(testcase)
2314- else:
2315- organizer[key] = [testcase]
2316- for value_list in organizer.values():
2317- ordered_list = ordered_list + value_list
2318- self.test_list = ordered_list
2319-
2320-
2321-
2322
2323=== removed directory 'dbqp/lib/modes/native'
2324=== removed file 'dbqp/lib/modes/native/__init__.py'
2325=== removed file 'dbqp/lib/modes/native/native_test_execution.py'
2326--- dbqp/lib/modes/native/native_test_execution.py 2012-02-04 01:22:23 +0000
2327+++ dbqp/lib/modes/native/native_test_execution.py 1970-01-01 00:00:00 +0000
2328@@ -1,100 +0,0 @@
2329-#! /usr/bin/env python
2330-# -*- mode: python; indent-tabs-mode: nil; -*-
2331-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
2332-#
2333-# Copyright (C) 2011 Patrick Crews
2334-#
2335-#
2336-# This program is free software; you can redistribute it and/or modify
2337-# it under the terms of the GNU General Public License as published by
2338-# the Free Software Foundation; either version 2 of the License, or
2339-# (at your option) any later version.
2340-#
2341-# This program is distributed in the hope that it will be useful,
2342-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2343-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2344-# GNU General Public License for more details.
2345-#
2346-# You should have received a copy of the GNU General Public License
2347-# along with this program; if not, write to the Free Software
2348-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2349-
2350-""" native_test_execution:
2351- code related to the execution of native test cases
2352-
2353- We are provided access to a testManager with
2354- native-specific testCases.
2355-
2356-"""
2357-
2358-# imports
2359-import os
2360-import re
2361-import imp
2362-import sys
2363-import subprocess
2364-import unittest
2365-
2366-import lib.test_mgmt.test_execution as test_execution
2367-
2368-class testExecutor(test_execution.testExecutor):
2369- """ native-mode-specific executor """
2370-
2371- def execute_testCase (self):
2372- """ Execute a test module testCase
2373-
2374- """
2375- test_execution.testExecutor.execute_testCase(self)
2376- self.status = 0
2377-
2378- # execute test module
2379- self.current_test_status = self.execute_test_module()
2380-
2381- # analyze results
2382- self.set_server_status(self.current_test_status)
2383- self.server_manager.reset_servers(self.name)
2384-
2385-
2386- def execute_test_module(self):
2387- """ Execute the commandline and return the result.
2388- We use subprocess as we can pass os.environ dicts and whatnot
2389-
2390- """
2391- output_file_path = os.path.join(self.logdir,'native.out')
2392- output_file = open(output_file_path,'w')
2393- testcase_name = self.current_testcase.fullname
2394- test_name = self.current_testcase.name
2395-
2396- # import our module and pass it some goodies to play with
2397- test_module = imp.load_source(test_name, self.current_testcase.test_path)
2398- test_module.servers = self.current_servers
2399- test_module.test_executor = self
2400- test_module.server_manager = self.server_manager
2401-
2402- # start our test
2403- self.time_manager.start(testcase_name,'test')
2404- self.logging.subunit_start(testcase_name)
2405- suite = unittest.TestLoader().loadTestsFromTestCase(test_module.basicTest)
2406- test_result = unittest.TextTestRunner(stream=output_file, verbosity=2).run(suite)
2407- execution_time = int(self.time_manager.stop(testcase_name)*1000) # millisec
2408- self.current_test_retcode = test_result.wasSuccessful()
2409- output_file.close()
2410- output_file = open(output_file_path,'r')
2411- self.current_test_output = ''.join(output_file.readlines())
2412- output_file.close()
2413-
2414- self.current_test_exec_time = execution_time
2415- retval = None
2416- if self.current_test_retcode:
2417- if not self.verbose:
2418- self.current_test_output = None
2419- retval = 'pass'
2420- else:
2421- retval = 'fail'
2422- self.logging.subunit_stop( testcase_name
2423- , retval
2424- , self.current_test_output
2425- )
2426- return retval
2427-
2428-
2429
2430=== removed file 'dbqp/lib/modes/native/native_test_management.py'
2431--- dbqp/lib/modes/native/native_test_management.py 2012-02-04 01:22:23 +0000
2432+++ dbqp/lib/modes/native/native_test_management.py 1970-01-01 00:00:00 +0000
2433@@ -1,160 +0,0 @@
2434-#! /usr/bin/env python
2435-# -*- mode: python; indent-tabs-mode: nil; -*-
2436-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
2437-#
2438-# Copyright (C) 2011 Patrick Crews
2439-#
2440-## This program is free software; you can redistribute it and/or modify
2441-# it under the terms of the GNU General Public License as published by
2442-# the Free Software Foundation; either version 2 of the License, or
2443-# (at your option) any later version.
2444-#
2445-# This program is distributed in the hope that it will be useful,
2446-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2447-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2448-# GNU General Public License for more details.
2449-#
2450-# You should have received a copy of the GNU General Public License
2451-# along with this program; if not, write to the Free Software
2452-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2453-
2454-""" crashme_test_management:
2455- code related to the gathering / analysis / management of
2456- the test cases
2457- ie - collecting the list of tests in each suite, then
2458- gathering additional, relevant information for crashme mode
2459-
2460-"""
2461-
2462-# imports
2463-import os
2464-import re
2465-import sys
2466-import imp
2467-
2468-import lib.test_mgmt.test_management as test_management
2469-
2470-
2471-
2472-class testCase:
2473- """Holds info on a single crashme test
2474-
2475- """
2476- def __init__( self
2477- , system_manager
2478- , name=None
2479- , fullname = None
2480- , server_requirements=[[]]
2481- , comment=None
2482- , cnf_path=None
2483- , request_dict=None
2484- , test_path = None
2485- , suitename = 'native_tests'
2486- , debug=False ):
2487- self.system_manager = system_manager
2488- self.logging = self.system_manager.logging
2489- self.skip_keys = ['system_manager','logging']
2490- self.name = name
2491- self.fullname = fullname
2492- self.suitename = suitename
2493- self.master_sh = None
2494- self.comment = comment
2495- self.server_requirements = server_requirements
2496- self.cnf_path = cnf_path
2497- self.server_requests = request_dict
2498- self.test_path = test_path
2499- if debug:
2500- self.system_manager.logging.debug_class(self)
2501-
2502- def should_run(self):
2503- if self.skip_flag or self.disable:
2504- return 0
2505- else:
2506- return 1
2507-
2508-
2509-
2510-
2511-
2512-class testManager(test_management.testManager):
2513- """Deals with scanning test directories, gathering test cases, and
2514- collecting per-test information (opt files, etc) for use by the
2515- test-runner
2516-
2517- """
2518-
2519- def __init__( self, variables, system_manager):
2520- super(testManager, self).__init__( variables, system_manager)
2521- server_type = variables['defaultservertype']
2522- if server_type == 'mysql' or server_type =='galera':
2523- server_type = 'percona'
2524- self.suitepaths = [os.path.join(self.testdir,'%s_tests' %(server_type))]
2525- if variables['suitelist'] is None:
2526- self.suitelist = ['main']
2527- else:
2528- self.suitelist = variables['suitelist']
2529-
2530- def process_suite(self,suite_dir):
2531- """Process a test suite.
2532- Look for tests, which are nice clean python unittest files
2533-
2534- """
2535-
2536- # We know this based on how we organize native test conf files
2537- suite_name = os.path.basename(suite_dir)
2538- self.system_manager.logging.verbose("Processing suite: %s" %(suite_name))
2539- testlist = [os.path.join(suite_dir,test_file) for test_file in sorted(os.listdir(suite_dir)) if test_file.endswith('_test.py')]
2540-
2541- # Search for specific test names
2542- if self.desired_tests: # We have specific, named tests we want from the suite(s)
2543- tests_to_use = []
2544- for test in self.desired_tests:
2545- if test.endswith('.py'):
2546- pass
2547- else:
2548- test = test+'.py'
2549- test = os.path.join(suite_dir,test)
2550- if test in testlist:
2551- tests_to_use.append(test)
2552- testlist = tests_to_use
2553- for test_case in testlist:
2554- self.add_test(self.process_test_file(suite_name, test_case))
2555-
2556-
2557- def process_test_file(self, suite_name, testfile):
2558- """ We convert the info in a testfile into a testCase object """
2559-
2560- # test_name = filename - .py...simpler
2561- test_name = os.path.basename(testfile).replace('.py','')
2562- test_comment = None
2563- test_module = imp.load_source(test_name, testfile)
2564- server_requirements = test_module.server_requirements
2565- try:
2566- server_requests = test_module.server_requests
2567- except AttributeError, NameError:
2568- server_requests = None
2569- return testCase( self.system_manager
2570- , name = test_name
2571- , fullname = "%s.%s" %(suite_name, test_name)
2572- , server_requirements = server_requirements
2573- , cnf_path = None
2574- , request_dict = server_requests
2575- , test_path = testfile
2576- , debug = self.debug )
2577-
2578-
2579-
2580- def record_test_result(self, test_case, test_status, output, exec_time):
2581- """ Accept the results of an executed testCase for further
2582- processing.
2583-
2584- """
2585- if test_status not in self.executed_tests:
2586- self.executed_tests[test_status] = [test_case]
2587- else:
2588- self.executed_tests[test_status].append(test_case)
2589- # report
2590- self.logging.test_report( test_case.fullname, test_status
2591- , str(exec_time), output
2592- , report_output= True)
2593-
2594
2595=== removed directory 'dbqp/lib/modes/sysbench'
2596=== removed file 'dbqp/lib/modes/sysbench/__init__.py'
2597=== removed file 'dbqp/lib/modes/sysbench/sysbench_test_execution.py'
2598--- dbqp/lib/modes/sysbench/sysbench_test_execution.py 2012-02-04 01:22:23 +0000
2599+++ dbqp/lib/modes/sysbench/sysbench_test_execution.py 1970-01-01 00:00:00 +0000
2600@@ -1,162 +0,0 @@
2601-#! /usr/bin/env python
2602-# -*- mode: python; indent-tabs-mode: nil; -*-
2603-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
2604-#
2605-# Copyright (C) 2010 Patrick Crews
2606-#
2607-#
2608-# This program is free software; you can redistribute it and/or modify
2609-# it under the terms of the GNU General Public License as published by
2610-# the Free Software Foundation; either version 2 of the License, or
2611-# (at your option) any later version.
2612-#
2613-# This program is distributed in the hope that it will be useful,
2614-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2615-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2616-# GNU General Public License for more details.
2617-#
2618-# You should have received a copy of the GNU General Public License
2619-# along with this program; if not, write to the Free Software
2620-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2621-
2622-""" sysbench_test_execution:
2623- code related to the execution of sysbench test cases
2624-
2625- We are provided access to a testManager with
2626- sysbench-specific testCases.
2627-
2628-"""
2629-
2630-# imports
2631-import os
2632-import re
2633-import sys
2634-import subprocess
2635-import commands
2636-
2637-import lib.test_mgmt.test_execution as test_execution
2638-
2639-class testExecutor(test_execution.testExecutor):
2640- """ sysbench-specific testExecutor
2641-
2642- """
2643-
2644- def execute_testCase (self):
2645- """ Execute a sysbench testCase
2646-
2647- """
2648- test_execution.testExecutor.execute_testCase(self)
2649- self.status = 0
2650-
2651- # prepare the server for sysbench
2652- self.prepare_sysbench()
2653-
2654- # execute sysbench
2655- self.execute_sysbench()
2656-
2657- # analyze results
2658- self.current_test_status = self.process_sysbench_output()
2659- self.set_server_status(self.current_test_status)
2660- self.server_manager.reset_servers(self.name)
2661-
2662- def prepare_sysbench(self):
2663- """ Prepare the server for a sysbench run
2664- We use subprocess as we can pass os.environ dicts and whatnot
2665-
2666- """
2667-
2668- sysbench_outfile = os.path.join(self.logdir,'sysbench.out')
2669- sysbench_output = open(sysbench_outfile,'w')
2670- sysbench_cmd = ' '.join([self.current_testcase.test_command,'prepare'])
2671- self.logging.info("Preparing database for sysbench run...")
2672- self.logging.debug(sysbench_cmd)
2673- sysbench_subproc = subprocess.Popen( sysbench_cmd
2674- , shell=True
2675- #, cwd=os.getcwd()
2676- , env=self.working_environment
2677- , stdout = sysbench_output
2678- , stderr = subprocess.STDOUT
2679- )
2680- sysbench_subproc.wait()
2681- retcode = sysbench_subproc.returncode
2682-
2683- sysbench_output.close()
2684- sysbench_file = open(sysbench_outfile,'r')
2685- output = ''.join(sysbench_file.readlines())
2686- sysbench_file.close()
2687- self.logging.debug("sysbench_retcode: %d" %(retcode))
2688- self.logging.debug(output)
2689- if retcode:
2690- self.logging.error("sysbench_prepare failed with retcode %d:" %(retcode))
2691- self.logging.error(output)
2692- sys.exit(1)
2693-
2694-
2695-
2696-
2697- def execute_sysbench(self):
2698- """ Execute the commandline and return the result.
2699- We use subprocess as we can pass os.environ dicts and whatnot
2700-
2701- """
2702-
2703- testcase_name = self.current_testcase.fullname
2704- self.time_manager.start(testcase_name,'test')
2705- sysbench_outfile = os.path.join(self.logdir,'sysbench.out')
2706- sysbench_output = open(sysbench_outfile,'w')
2707- sysbench_cmd = ' '.join([self.current_testcase.test_command, 'run'])
2708- self.logging.info("Executing sysbench: %s" %(sysbench_cmd))
2709-
2710- sysbench_subproc = subprocess.Popen( sysbench_cmd
2711- , shell=True
2712- #, cwd=self.system_manager.sysbench_path
2713- , env=self.working_environment
2714- , stdout = sysbench_output
2715- , stderr = subprocess.STDOUT
2716- )
2717- sysbench_subproc.wait()
2718- retcode = sysbench_subproc.returncode
2719- execution_time = int(self.time_manager.stop(testcase_name)*1000) # millisec
2720-
2721- sysbench_output.close()
2722- sysbench_file = open(sysbench_outfile,'r')
2723- output = ''.join(sysbench_file.readlines())
2724- self.logging.debug(output)
2725- sysbench_file.close()
2726-
2727- self.logging.debug("sysbench_retcode: %d" %(retcode))
2728- self.current_test_retcode = retcode
2729- self.current_test_output = output
2730- self.current_test_exec_time = execution_time
2731-
2732- def process_sysbench_output(self):
2733- """ sysbench has run, we now check out what we have
2734- We also output the data from the run
2735-
2736- """
2737- # This slice code taken from drizzle-automation's sysbench handling
2738- # Slice up the output report into a matrix and insert into the DB.
2739- regexes= {
2740- 'tps': re.compile(r".*transactions\:\s+\d+\D*(\d+\.\d+).*")
2741- , 'deadlocksps': re.compile(r".*deadlocks\:\s+\d+\D*(\d+\.\d+).*")
2742- , 'rwreqps': re.compile(r".*read\/write\s+requests\:\s+\d+\D*(\d+\.\d+).*")
2743- , 'min_req_lat_ms': re.compile(r".*min\:\s+(\d*\.\d+)ms.*")
2744- , 'max_req_lat_ms': re.compile(r".*max\:\s+(\d*\.\d+)ms.*")
2745- , 'avg_req_lat_ms': re.compile(r".*avg\:\s+(\d*\.\d+)ms.*")
2746- , '95p_req_lat_ms': re.compile(r".*approx.\s+95\s+percentile\:\s+(\d+\.\d+)ms.*")
2747- }
2748- run= {}
2749- for line in self.current_test_output.split("\n"):
2750- for key in regexes.keys():
2751- result= regexes[key].match(line)
2752- if result:
2753- run[key]= float(result.group(1)) # group(0) is entire match...
2754- # we set our test output to the regex'd-up data
2755- # we also make it a single string, separated by newlines
2756- self.current_test_output = str(run)[1:-1].replace(',','\n').replace("'",'')
2757-
2758- if self.current_test_retcode == 0:
2759- return 'pass'
2760- else:
2761- return 'fail'
2762-
2763
2764=== removed file 'dbqp/lib/modes/sysbench/sysbench_test_management.py'
2765--- dbqp/lib/modes/sysbench/sysbench_test_management.py 2012-02-04 01:22:23 +0000
2766+++ dbqp/lib/modes/sysbench/sysbench_test_management.py 1970-01-01 00:00:00 +0000
2767@@ -1,159 +0,0 @@
2768-#! /usr/bin/env python
2769-# -*- mode: python; indent-tabs-mode: nil; -*-
2770-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
2771-#
2772-# Copyright (C) 2010 Patrick Crews
2773-#
2774-## This program is free software; you can redistribute it and/or modify
2775-# it under the terms of the GNU General Public License as published by
2776-# the Free Software Foundation; either version 2 of the License, or
2777-# (at your option) any later version.
2778-#
2779-# This program is distributed in the hope that it will be useful,
2780-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2781-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2782-# GNU General Public License for more details.
2783-#
2784-# You should have received a copy of the GNU General Public License
2785-# along with this program; if not, write to the Free Software
2786-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2787-
2788-""" sysbench_test_management:
2789- code related to the gathering / analysis / management of
2790- the test cases
2791- ie - collecting the list of tests in each suite, then
2792- gathering additional, relevant information for sysbench mode
2793-
2794-"""
2795-
2796-# imports
2797-import os
2798-import re
2799-import sys
2800-from ConfigParser import RawConfigParser
2801-
2802-import lib.test_mgmt.test_management as test_management
2803-
2804-
2805-
2806-class testCase:
2807- """Holds info on a single sysbench test
2808-
2809- """
2810- def __init__( self, system_manager, name=None
2811- , fullname = None, server_requirements=[[]]
2812- , comment=None, test_command=None, cnf_path=None
2813- , debug=False ):
2814- self.system_manager = system_manager
2815- self.logging = self.system_manager.logging
2816- self.skip_keys = ['system_manager','logging']
2817- self.name = name
2818- self.fullname = fullname
2819- self.suitename = 'sysbench_tests'
2820- self.master_sh = None
2821- self.comment = comment
2822- self.server_requirements = server_requirements
2823- self.test_command = test_command
2824- self.cnf_path = cnf_path
2825-
2826- if debug:
2827- self.system_manager.logging.debug_class(self)
2828-
2829- def should_run(self):
2830- if self.skip_flag or self.disable:
2831- return 0
2832- else:
2833- return 1
2834-
2835-
2836-
2837-
2838-
2839-class testManager(test_management.testManager):
2840- """Deals with scanning test directories, gathering test cases, and
2841- collecting per-test information (opt files, etc) for use by the
2842- test-runner
2843-
2844- """
2845-
2846- def __init__( self, variables, system_manager):
2847- super(testManager, self).__init__( variables, system_manager)
2848- self.suitepaths = [os.path.join(self.testdir,'sysbench_tests')]
2849- if variables['suitelist'] is None:
2850- self.suitelist = ['readonly']
2851- else:
2852- self.suitelist = variables['suitelist']
2853-
2854- def process_suite(self,suite_dir):
2855- """Process a test suite.
2856- Look for sysbench tests, which are nice clean conf files
2857-
2858- """
2859-
2860- # We know this based on how we organize sysbench test conf files
2861- suite_name = os.path.basename(suite_dir)
2862- self.system_manager.logging.verbose("Processing suite: %s" %(suite_name))
2863- testlist = [os.path.join(suite_dir,test_file) for test_file in sorted(os.listdir(suite_dir)) if test_file.endswith('.cnf')]
2864-
2865- # Search for specific test names
2866- if self.desired_tests: # We have specific, named tests we want from the suite(s)
2867- tests_to_use = []
2868- for test in self.desired_tests:
2869- if test.endswith('.cnf'):
2870- pass
2871- else:
2872- test = test+'.cnf'
2873- test = os.path.join(suite_dir,test)
2874- if test in testlist:
2875- tests_to_use.append(test)
2876- testlist = tests_to_use
2877- for test_case in testlist:
2878- self.add_test(self.process_test_file(suite_name, test_case))
2879-
2880-
2881- def process_test_file(self, suite_name, testfile):
2882- """ We convert the info in a testfile into a testCase object """
2883-
2884- config_reader = RawConfigParser()
2885- config_reader.read(testfile)
2886- # test_name = filename - .cnf...simpler
2887- test_name = os.path.basename(testfile).replace('.cnf','')
2888- test_comment = config_reader.get('test_info','comment')
2889- server_requirements = self.process_server_reqs(config_reader.get('test_servers','servers'))
2890- test_command = config_reader.get('test_command','command')
2891- return testCase( self.system_manager
2892- , name = test_name
2893- , fullname = "%s.%s" %(suite_name, test_name)
2894- , server_requirements = server_requirements
2895- , test_command = test_command
2896- , cnf_path = testfile
2897- , debug = self.debug )
2898-
2899- #sys.exit(0)
2900-
2901- def process_server_reqs(self,data_string):
2902- """ We read in the list of lists as a string, so we need to
2903- handle this / break it down into proper chunks
2904-
2905- """
2906- server_reqs = []
2907- # We expect to see a list of lists and throw away the
2908- # enclosing brackets
2909- option_sets = data_string[1:-1].strip().split(',')
2910- for option_set in option_sets:
2911- server_reqs.append([option_set[1:-1].strip()])
2912- return server_reqs
2913-
2914- def record_test_result(self, test_case, test_status, output, exec_time):
2915- """ Accept the results of an executed testCase for further
2916- processing.
2917-
2918- """
2919- if test_status not in self.executed_tests:
2920- self.executed_tests[test_status] = [test_case]
2921- else:
2922- self.executed_tests[test_status].append(test_case)
2923- # report
2924- self.logging.test_report( test_case.fullname, test_status
2925- , str(exec_time), output
2926- , report_output= True)
2927
2928=== removed file 'dbqp/lib/modes/test_mode.py'
2929--- dbqp/lib/modes/test_mode.py 2012-02-04 01:22:23 +0000
2930+++ dbqp/lib/modes/test_mode.py 1970-01-01 00:00:00 +0000
2931@@ -1,68 +0,0 @@
2932-#! /usr/bin/env python
2933-# -*- mode: python; indent-tabs-mode: nil; -*-
2934-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
2935-#
2936-# Copyright (C) 2010 Patrick Crews
2937-#
2938-# This program is free software; you can redistribute it and/or modify
2939-# it under the terms of the GNU General Public License as published by
2940-# the Free Software Foundation; either version 2 of the License, or
2941-# (at your option) any later version.
2942-#
2943-# This program is distributed in the hope that it will be useful,
2944-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2945-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2946-# GNU General Public License for more details.
2947-#
2948-# You should have received a copy of the GNU General Public License
2949-# along with this program; if not, write to the Free Software
2950-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
2951-
2952-"""test_mode.py
2953- code for dealing with testing modes
2954- A given mode should have a systemInitializer, testManager, and testExecutor
2955- that define how to setup, manage, and execute test cases
2956-
2957-"""
2958-
2959-# imports
2960-import sys
2961-
2962-def handle_mode(variables, system_manager):
2963- """ Deals with the 'mode' option and returns
2964- the appropriate code objects for the test-runner to play with
2965-
2966- """
2967-
2968- test_mode = variables['mode'].strip()
2969- system_manager.logging.info("Using testing mode: %s" %test_mode)
2970-
2971- if test_mode == 'cleanup':
2972- # cleanup mode - we try to kill any servers whose pid's we detect
2973- # in our workdir. Might extend to other things (file cleanup, etc)
2974- # at some later point
2975- system_manager.cleanup(exit=True)
2976-
2977- else: # we expect something from dbqp_modes
2978- supported_modes = [ 'dtr'
2979- , 'randgen'
2980- , 'sysbench'
2981- , 'sqlbench'
2982- , 'crashme'
2983- , 'native'
2984- ]
2985- if test_mode not in supported_modes:
2986- system_manager.logging.error("invalid mode argument: %s" %test_mode)
2987- sys.exit(1)
2988-
2989- mgmt_module = "lib.modes.%s.%s_test_management" %(test_mode, test_mode)
2990- tmp = __import__(mgmt_module, globals(), locals(), ['testManager'], -1)
2991- testManager = tmp.testManager
2992-
2993- exec_module = "%s.%s_test_execution" %(test_mode, test_mode)
2994- tmp = __import__(exec_module, globals(), locals(), ['testExecutor'], -1)
2995- testExecutor = tmp.testExecutor
2996-
2997- test_manager = testManager( variables, system_manager )
2998- return (test_manager, testExecutor)
2999-
3000
3001=== removed directory 'dbqp/lib/opts'
3002=== removed file 'dbqp/lib/opts/__init__.py'
3003=== removed file 'dbqp/lib/opts/test_run_options.py'
3004--- dbqp/lib/opts/test_run_options.py 2012-02-04 01:22:23 +0000
3005+++ dbqp/lib/opts/test_run_options.py 1970-01-01 00:00:00 +0000
3006@@ -1,556 +0,0 @@
3007-#! /usr/bin/env python
3008-# -*- mode: python; indent-tabs-mode: nil; -*-
3009-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
3010-#
3011-# Copyright (C) 2010, 2011 Patrick Crews
3012-#
3013-# This program is free software; you can redistribute it and/or modify
3014-# it under the terms of the GNU General Public License as published by
3015-# the Free Software Foundation; either version 2 of the License, or
3016-# (at your option) any later version.
3017-#
3018-# This program is distributed in the hope that it will be useful,
3019-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3020-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3021-# GNU General Public License for more details.
3022-#
3023-# You should have received a copy of the GNU General Public License
3024-# along with this program; if not, write to the Free Software
3025-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
3026-
3027-
3028-
3029-"""Processes command line options for Drizzle test-runner"""
3030-
3031-import os
3032-import sys
3033-import copy
3034-import exceptions
3035-import optparse
3036-
3037-# functions
3038-def comma_list_split(option, opt, value, parser):
3039- """Callback for splitting input expected in list form"""
3040- cur_list = getattr(parser.values, option.dest,[])
3041- input_list = value.split(',')
3042- # this is a hack to work with make target - we
3043- # don't deal with a dangling ',' in our list
3044- if '' in input_list:
3045- input_list.remove('')
3046- if cur_list:
3047- value_list = cur_list + input_list
3048- else:
3049- value_list = input_list
3050- setattr(parser.values, option.dest, value_list)
3051-
3052-def get_abspath(option, opt, value, parser):
3053- """ Utility function to make sure we have absolute paths
3054- if the user supplies values
3055-
3056- """
3057- the_path = os.path.abspath(value)
3058- setattr(parser.values, option.dest, the_path)
3059-
3060-def organize_options(args, test_cases):
3061- """Put our arguments in a nice dictionary
3062- We use option.dest as dictionary key
3063- item = supplied input
3064- ['
3065- """
3066- variables = {}
3067- # we make a copy as the python manual on vars
3068- # says we shouldn't alter the dictionary returned
3069- # by vars() - could affect symbol table?
3070- variables = copy.copy(vars(args))
3071- variables['test_cases']= test_cases
3072- # This code should become a function once
3073- # enough thought has been given to it
3074- if variables['manualgdb']:
3075- variables['gdb']=True
3076- if variables['repeat'] <= 0:
3077- print "Setting --repeat=1. You chose a silly value that I will ignore :P"
3078- variables['repeat'] = 1
3079- if variables['mode'] == 'randgen' or variables['gendatafile']:
3080- print "Setting --no-secure-file-priv=True for randgen usage..."
3081- variables['nosecurefilepriv']=True
3082- if variables['mode'] == 'cleanup':
3083- print "Setting --start-dirty=True for cleanup mode..."
3084- variables['startdirty']=True
3085- if variables['libeatmydata'] and os.path.exists(variables['libeatmydatapath']):
3086- # We are using libeatmydata vs. shared mem for server speedup
3087- print "Using libeatmydata at %s. Setting --no-shm / not using shared memory for testing..." %(variables['libeatmydatapath'])
3088- variables['noshm']=True
3089- return variables
3090-
3091-def populate_defaults(variables, basedir_default):
3092- """ We fill in any default values that need
3093- to be put in post-parsing
3094-
3095- """
3096- if not variables['basedir']:
3097- # We populate this value with the default now
3098- # it allows us to have a default and have user
3099- # supplied opts to override them
3100- variables['basedir'].append(basedir_default)
3101- return variables
3102-
3103-def handle_user_opts(variables, basedir_default, testdir_default, suitepaths_default):
3104- """ Some variables are dependent upon default values
3105- We do the probably hacky thing of going through
3106- and updating them accordingly
3107-
3108- We make the assumption / decision that only
3109- the first basedir value supplied should
3110- be applicable when searching for tests
3111-
3112- """
3113- master_basedir = os.path.abspath(variables['basedir'][0].split(':type:')[0])
3114- if master_basedir != basedir_default:
3115- new_path = os.path.join(master_basedir, 'plugin')
3116- search_path = os.path.join(basedir_default,'plugin')
3117- tmp = variables['suitepaths']
3118- tmp[tmp.index(search_path)] = new_path
3119- variables['suitepaths'] = tmp
3120- if variables['testdir'] != testdir_default:
3121- new_path = os.path.join(variables['testdir'],'suite')
3122- search_path = os.path.join(testdir_default,'suite')
3123- tmp = variables['suitepaths']
3124- tmp[tmp.index(search_path)] = new_path
3125- variables['suitepaths'] = tmp
3126- return variables
3127-
3128-
3129-# Create the CLI option parser
3130-parser= optparse.OptionParser(version='%prog (database quality platform aka project steve austin) version 0.1.1')
3131-
3132-# set some default values
3133-testdir_default = os.path.abspath(os.getcwd())
3134-workdir_default = os.path.join(testdir_default,'workdir')
3135-clientbindir_default = os.path.abspath(os.path.join(testdir_default,'../client'))
3136-basedir_default = os.path.dirname(testdir_default)
3137-server_type_default = 'galera'
3138-valgrind_suppression_default = os.path.join(testdir_default,'valgrind.supp')
3139-suitepaths_default = [ os.path.join(basedir_default,'plugin')
3140- , os.path.join(testdir_default,'suite')
3141- ]
3142-randgen_path_default = os.path.join(testdir_default,'randgen')
3143-subunit_file_default = os.path.join(workdir_default,'test_results.subunit')
3144-
3145-
3146-config_control_group = optparse.OptionGroup(parser,
3147- "Configuration controls - allows you to specify a file with a number of options already specified")
3148-config_control_group.add_option(
3149- "--sys_config_file"
3150- , dest="sysconfigfilepath"
3151- , action='store'
3152- , default=None # We want to have a file that will be our default defaults file...
3153- , help="The file that specifies system configuration specs for dbqp to execute tests (not yet implemented)"
3154- )
3155-parser.add_option_group(config_control_group)
3156-
3157-
3158-system_control_group = optparse.OptionGroup(parser,
3159- "Options for the test-runner itself - defining the system under test and how to execute tests")
3160-
3161-system_control_group.add_option(
3162- "--force"
3163- , dest="force"
3164- , action="store_true"
3165- , default=False
3166- , help="Set this to continue test execution beyond the first failed test"
3167- )
3168-
3169-system_control_group.add_option(
3170- "--start-and-exit"
3171- , dest="startandexit"
3172- , action="store_true"
3173- , default=False
3174- , help="Spin up the server(s) for the first specified test then exit (will leave servers running)"
3175- )
3176-
3177-system_control_group.add_option(
3178- "--verbose"
3179- , dest="verbose"
3180- , action="store_true"
3181- , default = False
3182- , help="Produces extensive output about test-runner state. Distinct from --debug"
3183- )
3184-
3185-system_control_group.add_option(
3186- "--debug"
3187- , dest="debug"
3188- , action="store_true"
3189- , default = False
3190- , help="Provide internal-level debugging output. Distinct from --verbose"
3191- )
3192-
3193-system_control_group.add_option(
3194- "--mode"
3195- , dest="mode"
3196- , default="native"
3197- , help="Testing mode. We currently support dtr, randgen, sysbench, sqlbench, crashme and cleanup modes. See docs for further details about individual modes [%default]"
3198- )
3199-
3200-system_control_group.add_option(
3201- "--record"
3202- , dest="record"
3203- , action="store_true"
3204- , default=False
3205- , help="Record a testcase result (if the testing mode supports it) [%default]"
3206- )
3207-
3208-system_control_group.add_option(
3209- "--fast"
3210- , dest="fast"
3211- , action="store_true"
3212- , default=False
3213- , help="Don't try to cleanup from earlier runs (currently just a placeholder) [%default]"
3214- )
3215-
3216-parser.add_option_group(system_control_group)
3217-
3218-test_control_group = optparse.OptionGroup(parser,
3219- "Options for controlling which tests are executed")
3220-
3221-test_control_group.add_option(
3222- "--suite"
3223- , dest="suitelist"
3224- , type='string'
3225- , action="callback"
3226- , callback=comma_list_split
3227- , help="The name of the suite containing tests we want. Can accept comma-separated list (with no spaces). Additional --suite args are appended to existing list [autosearch]"
3228- )
3229-
3230-test_control_group.add_option(
3231- "--suitepath"
3232- , dest="suitepaths"
3233- , type='string'
3234- , action="append"
3235- , default = suitepaths_default
3236- , help="The path containing the suite(s) you wish to execute. Use one --suitepath for each suite you want to use. [%default]"
3237- )
3238-
3239-test_control_group.add_option(
3240- "--do-test"
3241- , dest="dotest"
3242- , type='string'
3243- , default = None
3244- , help="input can either be a prefix or a regex. Will only execute tests that match the provided pattern"
3245- )
3246-
3247-test_control_group.add_option(
3248- "--skip-test"
3249- , dest="skiptest"
3250- , type='string'
3251- , default = None
3252- , help = "input can either be a prefix or a regex. Will exclude tests that match the provided pattern"
3253- )
3254-
3255-test_control_group.add_option(
3256- "--reorder"
3257- , dest="reorder"
3258- , action="store_true"
3259- , default=False
3260- , help = "sort the testcases so that they are executed optimally for the given mode [%default]"
3261- )
3262-
3263-test_control_group.add_option(
3264- "--repeat"
3265- , dest="repeat"
3266- , type='int'
3267- , action="store"
3268- , default=1
3269- , help = "Run each test case the specified number of times. For a given sequence, the first test will be run n times, then the second, etc [%default]"
3270- )
3271-
3272-parser.add_option_group(test_control_group)
3273-
3274-# test subject control group
3275-# terrible name for options tht define the server / code
3276-# that is under test
3277-
3278-# find some default values
3279-# assume we are in-tree testing in general and operating from root/test(?)
3280-testdir_default = os.path.abspath(os.getcwd())
3281-
3282-basedir_default = os.path.split(testdir_default)[0]
3283-
3284-test_subject_control_group = optparse.OptionGroup(parser,
3285- "Options for defining the code that will be under test")
3286-
3287-test_subject_control_group.add_option(
3288- "--basedir"
3289- , dest="basedir"
3290- , type='string'
3291- , default = []
3292- , action="append"
3293- , help = "Pass this argument to signal to the test-runner that this is an in-tree test. We automatically set a number of variables relative to the argument (client-bindir, serverdir, testdir) [%basedir_default]"
3294- )
3295-
3296-test_subject_control_group.add_option(
3297- "--default-server-type"
3298- , dest="defaultservertype"
3299- , type='string'
3300- , default = server_type_default
3301- , action='store'
3302- , help = "Defines what we consider to be the default server type. We assume a server is default type unless specified otherwise. [%default]"
3303- )
3304-
3305-test_subject_control_group.add_option(
3306- "--serverdir"
3307- , dest="serverpath"
3308- , type='string'
3309- , action="callback"
3310- , callback=get_abspath
3311- , help = "Path to the server executable. [%default]"
3312- )
3313-
3314-test_subject_control_group.add_option(
3315- "--client-bindir"
3316- , dest="clientbindir"
3317- , type = 'string'
3318- , action="callback"
3319- , callback=get_abspath
3320- , help = "Path to the directory containing client program binaries for use in testing [%default]"
3321- )
3322-
3323-
3324-test_subject_control_group.add_option(
3325- "--default-storage-engine"
3326- , dest="defaultengine"
3327- , default = 'innodb'
3328- , help="Start drizzled using the specified engine [%default]"
3329- )
3330-
3331-
3332-parser.add_option_group(test_subject_control_group)
3333-# end test subject control group
3334-
3335-# environment options
3336-
3337-environment_control_group = optparse.OptionGroup(parser,
3338- "Options for defining the testing environment")
3339-
3340-environment_control_group.add_option(
3341- "--testdir"
3342- , dest="testdir"
3343- , type = 'string'
3344- , default = testdir_default
3345- , action="callback"
3346- , callback=get_abspath
3347- , help = "Path to the test dir, containing additional files for test execution. [%default]"
3348- )
3349-
3350-environment_control_group.add_option(
3351- "--workdir"
3352- , dest="workdir"
3353- , type='string'
3354- , default = workdir_default
3355- , action="callback"
3356- , callback=get_abspath
3357- , help = "Path to the directory test-run will use to store generated files and directories. [%default]"
3358- )
3359-
3360-environment_control_group.add_option(
3361- "--top-srcdir"
3362- , dest="topsrcdir"
3363- , type='string'
3364- , default = basedir_default
3365- , help = "build option [%default]"
3366- )
3367-
3368-environment_control_group.add_option(
3369- "--top-builddir"
3370- , dest="topbuilddir"
3371- , type='string'
3372- , default = basedir_default
3373- , help = "build option [%default]"
3374- )
3375-
3376-environment_control_group.add_option(
3377- "--no-shm"
3378- , dest="noshm"
3379- , action='store_true'
3380- , default=False
3381- , help = "By default, we symlink workdir to a location in shm. Use this flag to not symlink [%default]"
3382- )
3383-
3384-environment_control_group.add_option(
3385- "--libeatmydata"
3386- , dest="libeatmydata"
3387- , action='store_true'
3388- , default=False
3389- , help = "We use libeatmydata (if available) to disable fsyncs and speed up test execution. Implies --no-shm"
3390- )
3391-
3392-environment_control_group.add_option(
3393- "--libeatmydata-path"
3394- , dest="libeatmydatapath"
3395- , action='store'
3396- , default='/usr/local/lib/libeatmydata.so'
3397- , help = "Path to the libeatmydata install you want to use [%default]"
3398- )
3399-
3400-environment_control_group.add_option(
3401- "--start-dirty"
3402- , dest="startdirty"
3403- , action='store_true'
3404- , default=False
3405- , help = "Don't try to clean up working directories before test execution [%default]"
3406- )
3407-
3408-environment_control_group.add_option(
3409- "--no-secure-file-priv"
3410- , dest = "nosecurefilepriv"
3411- , action='store_true'
3412- , default=False
3413- , help = "Turn off the use of --secure-file-priv=vardir for started servers"
3414- )
3415-
3416-environment_control_group.add_option(
3417- "--randgen-path"
3418- , dest="randgenpath"
3419- , action='store'
3420- , default=randgen_path_default
3421- , help = "The path to a randgen installation that can be used to execute randgen-based tests"
3422- )
3423-
3424-environment_control_group.add_option(
3425- "--innobackupex-path"
3426- , dest="innobackupexpath"
3427- , action='store'
3428- , default=None
3429- , help = "The path to the innobackupex script that facilitates the use of Xtrabackup"
3430- )
3431-
3432-environment_control_group.add_option(
3433- "--xtrabackup-path"
3434- , dest="xtrabackuppath"
3435- , action='store'
3436- , default=None
3437- , help = "The path the xtrabackup binary to be tested"
3438- )
3439-
3440-environment_control_group.add_option(
3441- "--wsrep-provider-path"
3442- , dest="wsrepprovider"
3443- , action='store'
3444- , default='/usr/lib/galera/libgalera_smm.so'
3445- , help = "The path to a wsrep provider library for use with mysql"
3446- )
3447-
3448-environment_control_group.add_option(
3449- "--cluster-cnf"
3450- , dest="clustercnf"
3451- , action='store'
3452- , default=None
3453- , help = "The path to a config file defining a running cluster (node info)"
3454- )
3455-
3456-environment_control_group.add_option(
3457- "--subunit-outfile"
3458- , dest="subunitoutfile"
3459- , action='store'
3460- , default=subunit_file_default
3461- , help = "File path where subunit output will be logged [%default]"
3462- )
3463-
3464-parser.add_option_group(environment_control_group)
3465-# end environment control group
3466-
3467-option_passing_group = optparse.OptionGroup(parser,
3468- "Options to pass options on to the server")
3469-
3470-option_passing_group.add_option(
3471-"--drizzled"
3472- , dest="drizzledoptions"
3473- , type='string'
3474- , action='append'
3475- , default = []
3476- , help = "Pass additional options to the server. Will be passed to all servers for all tests (mostly for --start-and-exit)"
3477- )
3478-
3479-parser.add_option_group(option_passing_group)
3480-# end option passing group
3481-
3482-analysis_control_group = optparse.OptionGroup(parser,
3483- "Options for defining the tools we use for code analysis (valgrind, gprof, gcov, etc)")
3484-
3485-analysis_control_group.add_option(
3486- "--valgrind"
3487- , dest="valgrind"
3488- , action='store_true'
3489- , default = False
3490- , help = "Run drizzletest and drizzled executables using valgrind with default options [%default]"
3491- )
3492-
3493-analysis_control_group.add_option(
3494- "--valgrind-option"
3495- , dest="valgrindarglist"
3496- , type='string'
3497- , action="append"
3498- , help = "Pass an option to valgrind (overrides/removes default valgrind options)"
3499- )
3500-
3501-analysis_control_group.add_option(
3502- "--valgrind-suppressions"
3503- , dest="valgrindsuppressions"
3504- , type='string'
3505- , action='store'
3506- , default = valgrind_suppression_default
3507- , help = "Point at a valgrind suppression file [%default]"
3508- )
3509-
3510-analysis_control_group.add_option(
3511- "--helgrind"
3512- , dest="helgrind"
3513- , action='store_true'
3514- , default=False
3515- , help="Use the helgrind tool for valgrind. Implies / will auto-use --valgrind"
3516- )
3517-
3518-parser.add_option_group(analysis_control_group)
3519-
3520-debugger_control_group = optparse.OptionGroup(parser,
3521- "Options for controlling the use of debuggers with test execution")
3522-
3523-debugger_control_group.add_option(
3524- "--gdb"
3525- , dest="gdb"
3526- , action='store_true'
3527- , default=False
3528- , help="Start the drizzled server(s) in gdb"
3529- )
3530-
3531-debugger_control_group.add_option(
3532- "--manual-gdb"
3533- , dest="manualgdb"
3534- , action='store_true'
3535- , default=False
3536- , help="Allows you to start the drizzled server(s) in gdb manually (in another window, etc)"
3537- )
3538-
3539-parser.add_option_group(debugger_control_group)
3540-
3541-utility_group = optparse.OptionGroup(parser,
3542- "Options to call additional utilities such as datagen")
3543-
3544-utility_group.add_option(
3545- "--gendata"
3546- , dest="gendatafile"
3547- , action='store'
3548- , type='string'
3549- , default=None
3550- , help="Call the randgen's gendata utility to use the specified configuration file. This will populate the server prior to any test execution")
3551-
3552-parser.add_option_group(utility_group)
3553-
3554-# supplied will be those arguments matching an option,
3555-# and test_cases will be everything else
3556-(args, test_cases)= parser.parse_args()
3557-
3558-variables = {}
3559-variables = organize_options(args, test_cases)
3560-variables = populate_defaults(variables, basedir_default)
3561-variables = handle_user_opts(variables, basedir_default, testdir_default, suitepaths_default)
3562-
3563
3564=== removed directory 'dbqp/lib/server_mgmt'
3565=== removed file 'dbqp/lib/server_mgmt/__init__.py'
3566=== removed file 'dbqp/lib/server_mgmt/drizzled.py'
3567--- dbqp/lib/server_mgmt/drizzled.py 2012-02-04 01:22:23 +0000
3568+++ dbqp/lib/server_mgmt/drizzled.py 1970-01-01 00:00:00 +0000
3569@@ -1,227 +0,0 @@
3570-#! /usr/bin/env python
3571-# -*- mode: python; indent-tabs-mode: nil; -*-
3572-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
3573-#
3574-# Copyright (C) 2010,2011 Patrick Crews
3575-#
3576-# This program is free software; you can redistribute it and/or modify
3577-# it under the terms of the GNU General Public License as published by
3578-# the Free Software Foundation; either version 2 of the License, or
3579-# (at your option) any later version.
3580-#
3581-# This program is distributed in the hope that it will be useful,
3582-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3583-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3584-# GNU General Public License for more details.
3585-#
3586-# You should have received a copy of the GNU General Public License
3587-# along with this program; if not, write to the Free Software
3588-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
3589-
3590-
3591-""" drizzled.py: code to allow a serverManager
3592- to provision and start up a drizzled server object
3593- for test execution
3594-
3595-"""
3596-
3597-# imports
3598-import os
3599-from lib.server_mgmt.server import Server
3600-
3601-class drizzleServer(Server):
3602- """ represents a drizzle server, its possessions
3603- (datadir, ports, etc), and methods for controlling
3604- and querying it
3605-
3606- TODO: create a base server class that contains
3607- standard methods from which we can inherit
3608- Currently there are definitely methods / attr
3609- which are general
3610-
3611- """
3612-
3613- def __init__( self, name, server_manager, code_tree, default_storage_engine
3614- , server_options, requester, workdir_root):
3615- super(drizzleServer, self).__init__( name
3616- , server_manager
3617- , code_tree
3618- , default_storage_engine
3619- , server_options
3620- , requester
3621- , workdir_root)
3622- self.preferred_base_port = 9306
3623-
3624- # client files
3625- self.drizzledump = self.code_tree.drizzledump
3626- self.drizzle_client = self.code_tree.drizzle_client
3627- self.drizzleimport = self.code_tree.drizzleimport
3628- self.drizzleslap = self.code_tree.drizzleslap
3629- self.server_path = self.code_tree.drizzle_server
3630- self.drizzle_client_path = self.code_tree.drizzle_client
3631- self.schemawriter = self.code_tree.schemawriter
3632- self.trx_reader = self.code_tree.trx_reader
3633-
3634- # Get our ports
3635- self.port_block = self.system_manager.port_manager.get_port_block( self.name
3636- , self.preferred_base_port
3637- , 6 )
3638- self.master_port = self.port_block[0]
3639- self.drizzle_tcp_port = self.port_block[1]
3640- self.mc_port = self.port_block[2]
3641- self.pbms_port = self.port_block[3]
3642- self.rabbitmq_node_port = self.port_block[4]
3643- self.json_server_port = self.port_block[5]
3644-
3645- # Generate our working directories
3646- self.dirset = {'var_%s' %(self.name): {'std_data_ln':( os.path.join(self.code_tree.testdir,'std_data'))
3647- ,'log':None
3648- ,'run':None
3649- ,'tmp':None
3650- ,'master-data': {'local': { 'test':None
3651- , 'mysql':None
3652- }
3653- }
3654- }
3655- }
3656- self.workdir = self.system_manager.create_dirset( workdir_root
3657- , self.dirset)
3658- self.vardir = self.workdir
3659- self.tmpdir = os.path.join(self.vardir,'tmp')
3660- self.rundir = os.path.join(self.vardir,'run')
3661- self.logdir = os.path.join(self.vardir,'log')
3662- self.datadir = os.path.join(self.vardir,'master-data')
3663-
3664- self.error_log = os.path.join(self.logdir,('%s.err' %(self.name)))
3665- self.pid_file = os.path.join(self.rundir,('%s.pid' %(self.name)))
3666- self.socket_file = os.path.join(self.vardir, ('%s.sock' %(self.name)))
3667- self.timer_file = os.path.join(self.logdir,('timer'))
3668-
3669- # Do magic to create a config file for use with the slave
3670- # plugin
3671- self.slave_config_file = os.path.join(self.logdir,'slave.cnf')
3672- self.create_slave_config_file()
3673-
3674- self.snapshot_path = os.path.join(self.tmpdir,('snapshot_%s' %(self.master_port)))
3675- # We want to use --secure-file-priv = $vardir by default
3676- # but there are times / tools when we need to shut this off
3677- if self.no_secure_file_priv:
3678- self.secure_file_string = ''
3679- else:
3680- self.secure_file_string = "--secure-file-priv='%s'" %(self.vardir)
3681- self.user_string = '--user=root'
3682-
3683- self.initialize_databases()
3684- self.take_db_snapshot()
3685-
3686- self.logging.debug_class(self)
3687-
3688- def report(self):
3689- """ We print out some general useful info """
3690- report_values = [ 'name'
3691- , 'master_port'
3692- , 'drizzle_tcp_port'
3693- , 'mc_port'
3694- , 'pbms_port'
3695- , 'rabbitmq_node_port'
3696- , 'vardir'
3697- , 'status'
3698- ]
3699- self.logging.info("%s server:" %(self.owner))
3700- for key in report_values:
3701- value = vars(self)[key]
3702- self.logging.info("%s: %s" %(key.upper(), value))
3703-
3704- def get_start_cmd(self):
3705- """ Return the command string that will start up the server
3706- as desired / intended
3707-
3708- """
3709-
3710- server_args = [ self.process_server_options()
3711- , "--mysql-protocol.port=%d" %(self.master_port)
3712- , "--mysql-protocol.connect-timeout=60"
3713- , "--innodb.data-file-path=ibdata1:20M:autoextend"
3714- , "--sort-buffer-size=256K"
3715- , "--max-heap-table-size=1M"
3716- , "--mysql-unix-socket-protocol.path=%s" %(self.socket_file)
3717- , "--pid-file=%s" %(self.pid_file)
3718- , "--drizzle-protocol.port=%d" %(self.drizzle_tcp_port)
3719- , "--default-storage-engine=%s" %(self.default_storage_engine)
3720- , "--datadir=%s" %(self.datadir)
3721- , "--tmpdir=%s" %(self.tmpdir)
3722- , self.secure_file_string
3723- , self.user_string
3724- ]
3725-
3726- if self.gdb:
3727- server_args.append('--gdb')
3728- return self.system_manager.handle_gdb_reqs(self, server_args)
3729- else:
3730- return "%s %s %s & " % ( self.cmd_prefix
3731- , self.server_path
3732- , " ".join(server_args)
3733- )
3734-
3735-
3736- def get_stop_cmd(self):
3737- """ Return the command that will shut us down """
3738-
3739- return "%s --user=root --port=%d --connect-timeout=5 --silent --password= --shutdown " %(self.drizzle_client_path, self.master_port)
3740-
3741-
3742- def get_ping_cmd(self):
3743- """Return the command string that will
3744- ping / check if the server is alive
3745-
3746- """
3747-
3748- return "%s --ping --port=%d --user=root" % (self.drizzle_client_path, self.master_port)
3749-
3750- def is_started(self):
3751- """ Determine if the server is up and running -
3752- this may vary from server type to server type
3753-
3754- """
3755-
3756- # We experiment with waiting for a pid file to be created vs. pinging
3757- # This is what test-run.pl does and it helps us pass logging_stats tests
3758- # while not self.ping_server(server, quiet=True) and timer != timeout:
3759-
3760- return self.system_manager.find_path( [self.pid_file]
3761- , required=0)
3762-
3763- def create_slave_config_file(self):
3764- """ Create a config file suitable for use
3765- with the slave-plugin. This allows
3766- us to tie other servers in easily
3767-
3768- """
3769-
3770- config_data = [ "[master1]"
3771- , "master-host=127.0.0.1"
3772- , "master-port=%d" %self.master_port
3773- , "master-user=root"
3774- , "master-pass=''"
3775- , "max-reconnects=100"
3776- #, "seconds-between-reconnects=20"
3777- ]
3778- outfile = open(self.slave_config_file,'w')
3779- for line in config_data:
3780- outfile.write("%s\n" %(line))
3781- outfile.close()
3782-
3783-
3784-
3785-
3786-
3787-
3788-
3789-
3790-
3791-
3792-
3793-
3794-
3795-
3796-
3797
3798=== removed file 'dbqp/lib/server_mgmt/galera.py'
3799--- dbqp/lib/server_mgmt/galera.py 2012-02-04 01:22:23 +0000
3800+++ dbqp/lib/server_mgmt/galera.py 1970-01-01 00:00:00 +0000
3801@@ -1,334 +0,0 @@
3802-#! /usr/bin/env python
3803-# -*- mode: python; indent-tabs-mode: nil; -*-
3804-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
3805-#
3806-# Copyright (C) 2010,2011 Patrick Crews
3807-#
3808-# This program is free software; you can redistribute it and/or modify
3809-# it under the terms of the GNU General Public License as published by
3810-# the Free Software Foundation; either version 2 of the License, or
3811-# (at your option) any later version.
3812-#
3813-# This program is distributed in the hope that it will be useful,
3814-# but WITHOUT ANY WARRANTY; without even the implied warranty of
3815-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
3816-# GNU General Public License for more details.
3817-#
3818-# You should have received a copy of the GNU General Public License
3819-# along with this program; if not, write to the Free Software
3820-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
3821-
3822-
3823-""" mysqld.py: code to allow a serverManager
3824- to provision and start up a mysqld server object
3825- for test execution
3826-
3827-"""
3828-
3829-# imports
3830-import os
3831-import sys
3832-import time
3833-import subprocess
3834-
3835-from ConfigParser import RawConfigParser
3836-
3837-from lib.server_mgmt.server import Server
3838-
3839-class mysqlServer(Server):
3840- """ represents a mysql server, its possessions
3841- (datadir, ports, etc), and methods for controlling
3842- and querying it
3843-
3844- TODO: create a base server class that contains
3845- standard methods from which we can inherit
3846- Currently there are definitely methods / attr
3847- which are general
3848-
3849- """
3850-
3851- def __init__( self, name, server_manager, code_tree, default_storage_engine
3852- , server_options, requester, workdir_root):
3853- super(mysqlServer, self).__init__( name
3854- , server_manager
3855- , code_tree
3856- , default_storage_engine
3857- , server_options
3858- , requester
3859- , workdir_root)
3860- self.preferred_base_port = 9306
3861-
3862- # client files
3863- self.mysqldump = self.code_tree.mysqldump
3864- self.mysqladmin = self.code_tree.mysqladmin
3865- self.mysql_client = self.code_tree.mysql_client
3866- self.mysqlimport = self.code_tree.mysqlimport
3867- self.mysqlslap = self.code_tree.mysqlslap
3868- self.mysql_upgrade = self.code_tree.mysql_upgrade
3869- self.server_path = self.code_tree.mysql_server
3870- self.mysql_client_path = self.code_tree.mysql_client
3871-
3872- # galera-specific info
3873- self.wsrep_cluster_addr=''
3874-
3875- # important stuff
3876- self.langdir = self.code_tree.langdir
3877- self.charsetdir = self.code_tree.charsetdir
3878- self.bootstrap_file = self.code_tree.bootstrap_path
3879- self.bootstrap_cmd = None
3880-
3881- # Get our ports
3882- self.port_block = self.system_manager.port_manager.get_port_block( self.name
3883- , self.preferred_base_port
3884- , 3 )
3885- self.master_port = self.port_block[0]
3886- self.galera_listen_port = self.port_block[1]
3887- self.galera_recv_port = self.port_block[2]
3888-
3889- # Generate our working directories
3890- self.dirset = { 'var_%s' %(self.name): {'std_data_ln':( os.path.join(self.code_tree.testdir,'std_data'))
3891- ,'log':None
3892- ,'run':None
3893- ,'tmp':None
3894- ,'master-data': { 'test':None
3895- , 'mysql':None
3896- }
3897- }
3898- }
3899- self.workdir = self.system_manager.create_dirset( workdir_root
3900- , self.dirset)
3901- self.vardir = self.workdir
3902- self.tmpdir = os.path.join(self.vardir,'tmp')
3903- self.rundir = os.path.join(self.vardir,'run')
3904- self.logdir = os.path.join(self.vardir,'log')
3905- self.std_data = os.path.join(self.vardir,'std_data_ln')
3906- self.datadir = os.path.join(self.vardir,'master-data')
3907-
3908- self.error_log = os.path.join(self.logdir,('error.log' ))
3909- self.bootstrap_log = os.path.join(self.logdir,('bootstrap.log'))
3910- self.pid_file = os.path.join(self.rundir,('my.pid'))
3911- self.socket_file = os.path.join(self.vardir, ('my.sock'))
3912- self.timer_file = os.path.join(self.logdir,('timer'))
3913- self.general_log_file = os.path.join(self.logdir,'mysqld.log')
3914- self.slow_query_log_file = os.path.join(self.logdir,'mysqld-slow.log')
3915- self.cnf_file = os.path.join(self.vardir,'my.cnf')
3916-
3917- self.snapshot_path = os.path.join(self.tmpdir,('snapshot_%s' %(self.master_port)))
3918- # We want to use --secure-file-priv = $vardir by default
3919- # but there are times / tools when we need to shut this off
3920- if self.no_secure_file_priv:
3921- self.secure_file_string = ''
3922- else:
3923- self.secure_file_string = "--secure-file-priv='%s'" %(self.vardir)
3924- self.user_string = '--user=root'
3925-
3926- self.initialize_databases()
3927- self.take_db_snapshot()
3928-
3929- self.logging.debug_class(self)
3930-
3931-
3932- def report(self):
3933- """ We print out some general useful info """
3934- report_values = [ 'name'
3935- , 'master_port'
3936- , 'galera_listen_port'
3937- , 'galera_recv_port'
3938- , 'socket_file'
3939- , 'vardir'
3940- , 'status'
3941- ]
3942- self.logging.info("%s server:" %(self.owner))
3943- for key in report_values:
3944- value = vars(self)[key]
3945- self.logging.info("%s: %s" %(key.upper(), value))
3946-
3947- def initialize_databases(self):
3948- """ Do the voodoo required to have a working database setup.
3949- For MySQL, this is calling the server with the
3950- --bootstrap argument. We generate the bootstrap
3951- file during codeTree intialization as the file is standard for
3952- all MySQL servers that are spawned from a single codeTree
3953-
3954- """
3955-
3956- # generate the bootstrap startup command
3957- if not self.bootstrap_cmd:
3958- mysqld_args = [ "--no-defaults"
3959- , "--bootstrap"
3960- , "--basedir=%s" %(self.code_tree.basedir)
3961- , "--datadir=%s" %(self.datadir)
3962- , "--loose-skip-falcon"
3963- , "--loose-skip-ndbcluster"
3964- , "--tmpdir=%s" %(self.tmpdir)
3965- , "--core-file"
3966- , "--lc-messages-dir=%s" %(self.langdir)
3967- , "--character-sets-dir=%s" %(self.charsetdir)
3968- ]
3969- # We add server_path into the mix this way as we
3970- # may alter how we store / handle server args later
3971- mysqld_args.insert(0,self.server_path)
3972- self.bootstrap_cmd = " ".join(mysqld_args)
3973- # execute our command
3974- bootstrap_log = open(self.bootstrap_log,'w')
3975- # open our bootstrap file
3976- bootstrap_in = open(self.bootstrap_file,'r')
3977- bootstrap_subproc = subprocess.Popen( self.bootstrap_cmd
3978- , shell=True
3979- , stdin=bootstrap_in
3980- , stdout=bootstrap_log
3981- , stderr=bootstrap_log
3982- )
3983- bootstrap_subproc.wait()
3984- bootstrap_in.close()
3985- bootstrap_log.close()
3986- bootstrap_retcode = bootstrap_subproc.returncode
3987- if bootstrap_retcode:
3988- self.logging.error("Received retcode: %s executing command: %s"
3989- %(bootstrap_retcode, self.bootstrap_cmd))
3990- self.logging.error("Check the bootstrap log: %s" %(self.bootstrap_log))
3991- sys.exit(1)
3992-
3993-
3994- def get_start_cmd(self):
3995- """ Return the command string that will start up the server
3996- as desired / intended
3997-
3998- """
3999-
4000- # we need to do some wsrep voodoo for galera rpl to work
4001- self.ip_address = self.system_manager.get_ip_address()
4002- self.listen_addr_string = "?gmcast.listen_addr=tcp://127.0.0.1:%d" %(self.galera_listen_port)
4003- # test for wsrep_provider argument
4004- wsrep_provider_string =''
4005- if self.system_manager.wsrep_provider_path:
4006- wsrep_provider_string = "--wsrep_provider=%s" %(self.system_manager.wsrep_provider_path)
4007- # handle wsrep_cluster_address argument
4008- wsrep_cluster_string ="--wsrep_cluster_address='gcomm://%s%s'" %( self.wsrep_cluster_addr
4009- , self.listen_addr_string)
4010- for server_opt in self.server_options:
4011- if server_opt.startswith("--wsrep_cluster_address="):
4012- # we have a user-specified value, so we don't
4013- # try to insert something
4014- wsrep_cluster_string = ''
4015- # check to see if they have specified a listen port
4016- # we want to use our own, so we have a better idea of
4017- # what is going on
4018- if '?gmcast.listen_addr' in server_opt:
4019- pass
4020- else:
4021- server_opt = server_opt.strip()+ listen_addr
4022- server_args = [ self.process_server_options()
4023- , "%s" %(wsrep_cluster_string)
4024- , "%s" %(wsrep_provider_string)
4025- , "--wsrep_debug=ON"
4026- , "--wsrep_provider_options='ist.recv_addr=%s:%d'" %(self.ip_address, self.galera_recv_port)
4027- , "--wsrep_sst_receive_address='127.0.0.1:%d'" %( self.master_port)
4028- , "--wsrep_sst_auth='root:'"
4029- , "--wsrep_node_name='node%d'" %(self.galera_listen_port)
4030- , "--bind-address=0.0.0.0"
4031- , "--innodb_locks_unsafe_for_binlog=1"
4032- , "--open-files-limit=1024"
4033- , "--query-cache-size=0"
4034- , "--local-infile"
4035- , "--character-set-server=latin1"
4036- , "--connect-timeout=60"
4037- , "--log-bin-trust-function-creators=1"
4038- , "--key_buffer_size=1M"
4039- , "--sort_buffer=256K"
4040- , "--max_heap_table_size=1M"
4041- , "--loose-innodb_data_file_path=ibdata1:10M:autoextend"
4042- , "--loose-innodb_buffer_pool_size=8M"
4043- , "--loose-innodb_write_io_threads=2"
4044- , "--loose-innodb_read_io_threads=2"
4045- , "--loose-innodb_log_buffer_size=1M"
4046- , "--loose-innodb_log_file_size=5M"
4047- , "--loose-innodb_additional_mem_pool_size=1M"
4048- , "--loose-innodb_log_files_in_group=2"
4049- , "--loose-innodb_lock_wait_timeout=9999999"
4050- , "--slave-net-timeout=120"
4051- , "--log-bin=%s" %(os.path.join(self.logdir,"mysqld-bin"))
4052- , "--binlog-direct-non-transactional-updates"
4053- , "--general_log=1"
4054- , "--general_log_file=%s" %(self.general_log_file)
4055- , "--slow_query_log=1"
4056- , "--slow_query_log_file=%s" %(self.slow_query_log_file)
4057- , "--basedir=%s" %(self.code_tree.basedir)
4058- , "--datadir=%s" %(self.datadir)
4059- , "--tmpdir=%s" %(self.tmpdir)
4060- , "--character-sets-dir=%s" %(self.charsetdir)
4061- , "--lc-messages-dir=%s" %(self.langdir)
4062- , "--port=%d" %(self.master_port)
4063- , "--socket=%s" %(self.socket_file)
4064- , "--pid-file=%s" %(self.pid_file)
4065- , "--default-storage-engine=%s" %(self.default_storage_engine)
4066- # server-id maybe needs fixing, but we want to avoid
4067- # the server-id=0 and no replication thing...
4068- , "--server-id=%d" %(self.get_numeric_server_id()+1)
4069- , self.secure_file_string
4070- , self.user_string
4071- ]
4072- self.gen_cnf_file(server_args)
4073-
4074- if self.gdb:
4075- server_args.append('--gdb')
4076- return self.system_manager.handle_gdb_reqs(self, server_args)
4077- else:
4078- return "%s %s %s & " % ( self.cmd_prefix
4079- , self.server_path
4080- , " ".join(server_args)
4081- )
4082-
4083-
4084- def get_stop_cmd(self):
4085- """ Return the command that will shut us down """
4086-
4087- return "%s --no-defaults --user=root --port=%d --host=127.0.0.1 --protocol=tcp shutdown " %(self.mysqladmin, self.master_port)
4088-
4089-
4090- def get_ping_cmd(self):
4091- """Return the command string that will
4092- ping / check if the server is alive
4093-
4094- """
4095-
4096- return '%s --no-defaults --user=root --port=%d --host=127.0.0.1 --protocol=tcp ping' % (self.mysqladmin, self.master_port)
4097-
4098- def is_started(self):
4099- """ Determine if the server is up and running -
4100- this may vary from server type to server type
4101-
4102- """
4103-
4104- # We experiment with waiting for a pid file to be created vs. pinging
4105- # This is what test-run.pl does and it helps us pass logging_stats tests
4106- # while not self.ping_server(server, quiet=True) and timer != timeout:
4107-
4108- return self.system_manager.find_path( [self.pid_file]
4109- , required=0)
4110-
4111- def gen_cnf_file(self, server_args):
4112- """ We generate a .cnf file for the server based
4113- on the arguments. We currently don't use
4114- this for much, but xtrabackup uses it, so
4115- we must produce one. This could also be
4116- helpful for testing / etc
4117-
4118- """
4119-
4120- config_file = open(self.cnf_file,'w')
4121- config_file.write('[mysqld]')
4122- for server_arg in server_args:
4123- # We currently have a list of string values
4124- # We need to remove any '--' stuff
4125- server_arg = server_arg.replace('--','')+'\n'
4126- config_file.write(server_arg)
4127- config_file.close()
4128-
4129- def set_master(self, master_server):
4130- """ We update things so that we use the provided master_server
4131- as our reference point for a new cluster / etc
4132-
4133- """
4134-
4135- self.wsrep_cluster_addr = '127.0.0.1:%d' %(master_server.galera_listen_port)
4136
4137=== removed file 'dbqp/lib/server_mgmt/mysqld.py'
4138--- dbqp/lib/server_mgmt/mysqld.py 2012-02-04 01:22:23 +0000
4139+++ dbqp/lib/server_mgmt/mysqld.py 1970-01-01 00:00:00 +0000
4140@@ -1,295 +0,0 @@
4141-#! /usr/bin/env python
4142-# -*- mode: python; indent-tabs-mode: nil; -*-
4143-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
4144-#
4145-# Copyright (C) 2010,2011 Patrick Crews
4146-#
4147-# This program is free software; you can redistribute it and/or modify
4148-# it under the terms of the GNU General Public License as published by
4149-# the Free Software Foundation; either version 2 of the License, or
4150-# (at your option) any later version.
4151-#
4152-# This program is distributed in the hope that it will be useful,
4153-# but WITHOUT ANY WARRANTY; without even the implied warranty of
4154-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4155-# GNU General Public License for more details.
4156-#
4157-# You should have received a copy of the GNU General Public License
4158-# along with this program; if not, write to the Free Software
4159-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
4160-
4161-
4162-""" mysqld.py: code to allow a serverManager
4163- to provision and start up a mysqld server object
4164- for test execution
4165-
4166-"""
4167-
4168-# imports
4169-import os
4170-import sys
4171-import time
4172-import subprocess
4173-
4174-from ConfigParser import RawConfigParser
4175-
4176-from lib.server_mgmt.server import Server
4177-
4178-class mysqlServer(Server):
4179- """ represents a mysql server, its possessions
4180- (datadir, ports, etc), and methods for controlling
4181- and querying it
4182-
4183- TODO: create a base server class that contains
4184- standard methods from which we can inherit
4185- Currently there are definitely methods / attr
4186- which are general
4187-
4188- """
4189-
4190- def __init__( self, name, server_manager, code_tree, default_storage_engine
4191- , server_options, requester, workdir_root):
4192- super(mysqlServer, self).__init__( name
4193- , server_manager
4194- , code_tree
4195- , default_storage_engine
4196- , server_options
4197- , requester
4198- , workdir_root)
4199- self.preferred_base_port = 9306
4200-
4201- # client files
4202- self.mysqldump = self.code_tree.mysqldump
4203- self.mysqladmin = self.code_tree.mysqladmin
4204- self.mysql_client = self.code_tree.mysql_client
4205- self.mysqlimport = self.code_tree.mysqlimport
4206- self.mysqlslap = self.code_tree.mysqlslap
4207- self.mysql_upgrade = self.code_tree.mysql_upgrade
4208- self.server_path = self.code_tree.mysql_server
4209- self.mysql_client_path = self.code_tree.mysql_client
4210-
4211- # important stuff
4212- self.langdir = self.code_tree.langdir
4213- self.charsetdir = self.code_tree.charsetdir
4214- self.bootstrap_file = self.code_tree.bootstrap_path
4215- self.bootstrap_cmd = None
4216-
4217- # Get our ports
4218- self.port_block = self.system_manager.port_manager.get_port_block( self.name
4219- , self.preferred_base_port
4220- , 1 )
4221- self.master_port = self.port_block[0]
4222-
4223- # Generate our working directories
4224- self.dirset = { 'var_%s' %(self.name): {'std_data_ln':( os.path.join(self.code_tree.testdir,'std_data'))
4225- ,'log':None
4226- ,'run':None
4227- ,'tmp':None
4228- ,'master-data': { 'test':None
4229- , 'mysql':None
4230- }
4231- }
4232- }
4233- self.workdir = self.system_manager.create_dirset( workdir_root
4234- , self.dirset)
4235- self.vardir = self.workdir
4236- self.tmpdir = os.path.join(self.vardir,'tmp')
4237- self.rundir = os.path.join(self.vardir,'run')
4238- self.logdir = os.path.join(self.vardir,'log')
4239- self.std_data = os.path.join(self.vardir,'std_data_ln')
4240- self.datadir = os.path.join(self.vardir,'master-data')
4241-
4242- self.error_log = os.path.join(self.logdir,('%s.err' %(self.name)))
4243- self.bootstrap_log = os.path.join(self.logdir,('bootstrap.log'))
4244- self.pid_file = os.path.join(self.rundir,('%s.pid' %(self.name)))
4245- self.socket_file = os.path.join(self.vardir, ('%s.sock' %(self.name)))
4246- self.timer_file = os.path.join(self.logdir,('timer'))
4247- self.general_log_file = os.path.join(self.logdir,'mysqld.log')
4248- self.slow_query_log_file = os.path.join(self.logdir,'mysqld-slow.log')
4249- self.cnf_file = os.path.join(self.vardir,'my.cnf')
4250-
4251- self.snapshot_path = os.path.join(self.tmpdir,('snapshot_%s' %(self.master_port)))
4252- # We want to use --secure-file-priv = $vardir by default
4253- # but there are times / tools when we need to shut this off
4254- if self.no_secure_file_priv:
4255- self.secure_file_string = ''
4256- else:
4257- self.secure_file_string = "--secure-file-priv='%s'" %(self.vardir)
4258- self.user_string = '--user=root'
4259-
4260- self.initialize_databases()
4261- self.take_db_snapshot()
4262-
4263- self.logging.debug_class(self)
4264-
4265-
4266- def report(self):
4267- """ We print out some general useful info """
4268- report_values = [ 'name'
4269- , 'master_port'
4270- , 'socket_file'
4271- , 'vardir'
4272- , 'status'
4273- ]
4274- self.logging.info("%s server:" %(self.owner))
4275- for key in report_values:
4276- value = vars(self)[key]
4277- self.logging.info("%s: %s" %(key.upper(), value))
4278-
4279- def initialize_databases(self):
4280- """ Do the voodoo required to have a working database setup.
4281- For MySQL, this is calling the server with the
4282- --bootstrap argument. We generate the bootstrap
4283- file during codeTree intialization as the file is standard for
4284- all MySQL servers that are spawned from a single codeTree
4285-
4286- """
4287-
4288- # generate the bootstrap startup command
4289- if not self.bootstrap_cmd:
4290- mysqld_args = [ "--no-defaults"
4291- , "--bootstrap"
4292- , "--basedir=%s" %(self.code_tree.basedir)
4293- , "--datadir=%s" %(self.datadir)
4294- , "--loose-skip-falcon"
4295- , "--loose-skip-ndbcluster"
4296- , "--tmpdir=%s" %(self.tmpdir)
4297- , "--core-file"
4298- , "--lc-messages-dir=%s" %(self.langdir)
4299- , "--character-sets-dir=%s" %(self.charsetdir)
4300- ]
4301- # We add server_path into the mix this way as we
4302- # may alter how we store / handle server args later
4303- mysqld_args.insert(0,self.server_path)
4304- self.bootstrap_cmd = " ".join(mysqld_args)
4305- # execute our command
4306- bootstrap_log = open(self.bootstrap_log,'w')
4307- # open our bootstrap file
4308- bootstrap_in = open(self.bootstrap_file,'r')
4309- bootstrap_subproc = subprocess.Popen( self.bootstrap_cmd
4310- , shell=True
4311- , stdin=bootstrap_in
4312- , stdout=bootstrap_log
4313- , stderr=bootstrap_log
4314- )
4315- bootstrap_subproc.wait()
4316- bootstrap_in.close()
4317- bootstrap_log.close()
4318- bootstrap_retcode = bootstrap_subproc.returncode
4319- if bootstrap_retcode:
4320- self.logging.error("Received retcode: %s executing command: %s"
4321- %(bootstrap_retcode, self.bootstrap_cmd))
4322- self.logging.error("Check the bootstrap log: %s" %(self.bootstrap_log))
4323- sys.exit(1)
4324-
4325-
4326- def get_start_cmd(self):
4327- """ Return the command string that will start up the server
4328- as desired / intended
4329-
4330- """
4331-
4332- server_args = [ self.process_server_options()
4333- , "--open-files-limit=1024"
4334- , "--local-infile"
4335- , "--character-set-server=latin1"
4336- , "--connect-timeout=60"
4337- , "--log-bin-trust-function-creators=1"
4338- , "--key_buffer_size=1M"
4339- , "--sort_buffer=256K"
4340- , "--max_heap_table_size=1M"
4341- , "--loose-innodb_data_file_path=ibdata1:10M:autoextend"
4342- , "--loose-innodb_buffer_pool_size=8M"
4343- , "--loose-innodb_write_io_threads=2"
4344- , "--loose-innodb_read_io_threads=2"
4345- , "--loose-innodb_log_buffer_size=1M"
4346- , "--loose-innodb_log_file_size=5M"
4347- , "--loose-innodb_additional_mem_pool_size=1M"
4348- , "--loose-innodb_log_files_in_group=2"
4349- , "--slave-net-timeout=120"
4350- , "--log-bin=%s" %(os.path.join(self.logdir,"mysqld-bin"))
4351- , "--loose-enable-performance-schema"
4352- , "--loose-performance-schema-max-mutex-instances=10000"
4353- , "--loose-performance-schema-max-rwlock-instances=10000"
4354- , "--loose-performance-schema-max-table-instances=500"
4355- , "--loose-performance-schema-max-table-handles=1000"
4356- , "--binlog-direct-non-transactional-updates"
4357- , "--loose-enable-performance-schema"
4358- , "--general_log=1"
4359- , "--general_log_file=%s" %(self.general_log_file)
4360- , "--slow_query_log=1"
4361- , "--slow_query_log_file=%s" %(self.slow_query_log_file)
4362- , "--basedir=%s" %(self.code_tree.basedir)
4363- , "--datadir=%s" %(self.datadir)
4364- , "--tmpdir=%s" %(self.tmpdir)
4365- , "--character-sets-dir=%s" %(self.charsetdir)
4366- , "--lc-messages-dir=%s" %(self.langdir)
4367- , "--ssl-ca=%s" %(os.path.join(self.std_data,'cacert.pem'))
4368- , "--ssl-cert=%s" %(os.path.join(self.std_data,'server-cert.pem'))
4369- , "--ssl-key=%s" %(os.path.join(self.std_data,'server-key.pem'))
4370- , "--port=%d" %(self.master_port)
4371- , "--socket=%s" %(self.socket_file)
4372- , "--pid-file=%s" %(self.pid_file)
4373- , "--default-storage-engine=%s" %(self.default_storage_engine)
4374- # server-id maybe needs fixing, but we want to avoid
4375- # the server-id=0 and no replication thing...
4376- , "--server-id=%d" %(self.get_numeric_server_id()+1)
4377- , self.secure_file_string
4378- , self.user_string
4379- ]
4380- self.gen_cnf_file(server_args)
4381-
4382- if self.gdb:
4383- server_args.append('--gdb')
4384- return self.system_manager.handle_gdb_reqs(self, server_args)
4385- else:
4386- return "%s %s %s & " % ( self.cmd_prefix
4387- , self.server_path
4388- , " ".join(server_args)
4389- )
4390-
4391-
4392- def get_stop_cmd(self):
4393- """ Return the command that will shut us down """
4394-
4395- return "%s --no-defaults --user=root --port=%d --host=127.0.0.1 --protocol=tcp shutdown " %(self.mysqladmin, self.master_port)
4396-
4397-
4398- def get_ping_cmd(self):
4399- """Return the command string that will
4400- ping / check if the server is alive
4401-
4402- """
4403-
4404- return '%s --no-defaults --user=root --port=%d --host=127.0.0.1 --protocol=tcp ping' % (self.mysqladmin, self.master_port)
4405-
4406- def is_started(self):
4407- """ Determine if the server is up and running -
4408- this may vary from server type to server type
4409-
4410- """
4411-
4412- # We experiment with waiting for a pid file to be created vs. pinging
4413- # This is what test-run.pl does and it helps us pass logging_stats tests
4414- # while not self.ping_server(server, quiet=True) and timer != timeout:
4415-
4416- return self.system_manager.find_path( [self.pid_file]
4417- , required=0)
4418-
4419- def gen_cnf_file(self, server_args):
4420- """ We generate a .cnf file for the server based
4421- on the arguments. We currently don't use
4422- this for much, but xtrabackup uses it, so
4423- we must produce one. This could also be
4424- helpful for testing / etc
4425-
4426- """
4427-
4428- config_file = open(self.cnf_file,'w')
4429- config_file.write('[mysqld]')
4430- for server_arg in server_args:
4431- # We currently have a list of string values
4432- # We need to remove any '--' stuff
4433- server_arg = server_arg.replace('--','')+'\n'
4434- config_file.write(server_arg)
4435- config_file.close()
4436
4437=== removed file 'dbqp/lib/server_mgmt/percona.py'
4438--- dbqp/lib/server_mgmt/percona.py 2012-02-04 01:22:23 +0000
4439+++ dbqp/lib/server_mgmt/percona.py 1970-01-01 00:00:00 +0000
4440@@ -1,273 +0,0 @@
4441-#! /usr/bin/env python
4442-# -*- mode: python; indent-tabs-mode: nil; -*-
4443-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
4444-#
4445-# Copyright (C) 2010,2011 Patrick Crews
4446-#
4447-# This program is free software; you can redistribute it and/or modify
4448-# it under the terms of the GNU General Public License as published by
4449-# the Free Software Foundation; either version 2 of the License, or
4450-# (at your option) any later version.
4451-#
4452-# This program is distributed in the hope that it will be useful,
4453-# but WITHOUT ANY WARRANTY; without even the implied warranty of
4454-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4455-# GNU General Public License for more details.
4456-#
4457-# You should have received a copy of the GNU General Public License
4458-# along with this program; if not, write to the Free Software
4459-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
4460-
4461-
4462-""" mysqld.py: code to allow a serverManager
4463- to provision and start up a mysqld server object
4464- for test execution
4465-
4466-"""
4467-
4468-# imports
4469-import os
4470-import sys
4471-import subproc
4472-
4473-
4474-from lib.server_mgmt.server import Server
4475-
4476-class mysqlServer(Server):
4477- """ represents a mysql server, its possessions
4478- (datadir, ports, etc), and methods for controlling
4479- and querying it
4480-
4481- TODO: create a base server class that contains
4482- standard methods from which we can inherit
4483- Currently there are definitely methods / attr
4484- which are general
4485-
4486- """
4487-
4488- def __init__( self, name, server_manager, code_tree, default_storage_engine
4489- , server_options, requester, workdir_root):
4490- super(mysqlServer, self).__init__( name
4491- , server_manager
4492- , code_tree
4493- , default_storage_engine
4494- , server_options
4495- , requester
4496- , workdir_root)
4497- self.preferred_base_port = 9306
4498-
4499- # client files
4500- self.mysqldump = self.code_tree.mysqldump
4501- self.mysqladmin = self.code_tree.mysqladmin
4502- self.mysql_client = self.code_tree.mysql_client
4503- self.mysqlimport = self.code_tree.mysqlimport
4504- self.mysqlslap = self.code_tree.mysqlslap
4505- self.server_path = self.code_tree.mysql_server
4506- self.mysql_client_path = self.code_tree.mysql_client
4507-
4508- # important stuff
4509- self.langdir = self.code_tree.langdir
4510- self.charsetdir = self.code_tree.charsetdir
4511- self.bootstrap_file = self.code_tree.bootstrap_path
4512- self.bootstrap_cmd = None
4513-
4514- # Get our ports
4515- self.port_block = self.system_manager.port_manager.get_port_block( self.name
4516- , self.preferred_base_port
4517- , 1 )
4518- self.master_port = self.port_block[0]
4519-
4520- # Generate our working directories
4521- self.dirset = { self.name : { 'var': {'std_data_ln':( os.path.join(self.code_tree.testdir,'std_data'))
4522- ,'log':None
4523- ,'run':None
4524- ,'tmp':None
4525- ,'master-data': {'local': { 'test':None
4526- , 'mysql':None
4527- }
4528- }
4529- }
4530- }
4531- }
4532- self.workdir = self.system_manager.create_dirset( workdir_root
4533- , self.dirset)
4534- self.vardir = os.path.join(self.workdir,'var')
4535- self.tmpdir = os.path.join(self.vardir,'tmp')
4536- self.rundir = os.path.join(self.vardir,'run')
4537- self.logdir = os.path.join(self.vardir,'log')
4538- self.std_data = os.path.join(self.vardir,'std_data_ln')
4539- self.datadir = os.path.join(self.vardir,'master-data')
4540-
4541- self.error_log = os.path.join(self.logdir,('%s.err' %(self.name)))
4542- self.bootstrap_log = os.path.join(self.logdir,('bootstrap.log'))
4543- self.pid_file = os.path.join(self.rundir,('%s.pid' %(self.name)))
4544- self.socket_file = os.path.join(self.vardir, ('%s.sock' %(self.name)))
4545- self.timer_file = os.path.join(self.logdir,('timer'))
4546- self.general_log_file = os.path.join(self.logdir,'mysqld.log')
4547- self.slow_query_log_file = os.path.join(self.logdir,'mysqld-slow.log')
4548-
4549- self.snapshot_path = os.path.join(self.tmpdir,('snapshot_%s' %(self.master_port)))
4550- # We want to use --secure-file-priv = $vardir by default
4551- # but there are times / tools when we need to shut this off
4552- if self.no_secure_file_priv:
4553- self.secure_file_string = ''
4554- else:
4555- self.secure_file_string = "--secure-file-priv='%s'" %(self.vardir)
4556- self.user_string = '--user=root'
4557-
4558- self.initialize_databases()
4559- self.take_db_snapshot()
4560-
4561- self.logging.debug_class(self)
4562-
4563- def report(self):
4564- """ We print out some general useful info """
4565- report_values = [ 'name'
4566- , 'master_port'
4567- , 'socket_file'
4568- , 'vardir'
4569- , 'status'
4570- ]
4571- self.logging.info("%s server:" %(self.owner))
4572- for key in report_values:
4573- value = vars(self)[key]
4574- self.logging.info("%s: %s" %(key.upper(), value))
4575-
4576- def initialize_databases(self):
4577- """ Do the voodoo required to have a working database setup.
4578- For MySQL, this is calling the server with the
4579- --bootstrap argument. We generate the bootstrap
4580- file during codeTree intialization as the file is standard for
4581- all MySQL servers that are spawned from a single codeTree
4582-
4583- """
4584-
4585- # generate the bootstrap startup command
4586- if not self.bootstrap_cmd:
4587- mysqld_args = [ "--no-defaults"
4588- , "--bootstrap"
4589- , "--basedir=%s" %(self.code_tree.basedir)
4590- , "--datadir=%s" %(self.datadir)
4591- , "--loose-skip-falcon"
4592- , "--loose-skip-ndbcluster"
4593- , "--tmpdir=%s" %(self.tmpdir)
4594- , "--core-file"
4595- , "--lc-messages-dir=%s" %(self.langdir)
4596- , "--character-sets-dir=%s" %(self.charsetdir)
4597- ]
4598- # We add server_path into the mix this way as we
4599- # may alter how we store / handle server args later
4600- mysqld_args = [self.server_path].append(mysqld_args)
4601- self.bootstrap_cmd = " ".join(mysqld_args)
4602- # execute our command
4603- bootstrap_subproc = subprocess.Popen( self.bootstrap_cmd
4604- , shell=True
4605- , stdout=self.bootstrap_log
4606- , stderr=self.bootstrap_log
4607- )
4608- bootstrap_subproc.wait()
4609- bootstrap_retcode = bootstrap_subproc.returncode
4610- if bootstrap_retcode:
4611- self.logging.error("Received retcode: %s executing command: %s"
4612- %(bootstrap_retcode, self.bootstrap_cmd))
4613- self.logging.error("Check the bootstrap log: %s" %(self.bootstrap_log))
4614- sys.exit(1)
4615-
4616-
4617- def get_start_cmd(self):
4618- """ Return the command string that will start up the server
4619- as desired / intended
4620-
4621- """
4622-
4623- server_args = [ self.process_server_options()
4624- , "--open-files-limit=1024"
4625- , "--local-infile"
4626- , "--character-set-server=latin1"
4627- , "--connect-timeout=60"
4628- , "--log-bin-trust-function-creators=1"
4629- , "--key_buffer_size=1M"
4630- , "--sort_buffer=256K"
4631- , "--max_heap_table_size=1M"
4632- , "--query-cache-size=0"
4633- , "--loose-innodb_data_file_path=ibdata1:10M:autoextend"
4634- , "--loose-innodb_buffer_pool_size=32M"
4635- , "--loose-innodb_write_io_threads=2"
4636- , "--loose-innodb_read_io_threads=2"
4637- , "--loose-innodb_log_buffer_size=1M"
4638- , "--loose-innodb_log_file_size=5M"
4639- , "--loose-innodb_additional_mem_pool_size=1M"
4640- , "--loose-innodb_log_files_in_group=2"
4641- , "--slave-net-timeout=120"
4642- , "--log-bin=mysqld-bin"
4643- , "--loose-enable-performance-schema"
4644- , "--loose-performance-schema-max-mutex-instances=10000"
4645- , "--loose-performance-schema-max-rwlock-instances=10000"
4646- , "--loose-performance-schema-max-table-instances=500"
4647- , "--loose-performance-schema-max-table-handles=1000"
4648- , "--binlog-direct-non-transactional-updates"
4649- , "--loose-enable-performance-schema"
4650- , "--log-bin=master-bin"
4651- , "--general_log=1"
4652- , "--general_log_file=%s" %(self.general_log_file)
4653- , "--slow_query_log=1"
4654- , "--slow_query_log_file=%s" %(self.slow_query_log_file)
4655- , "--basedir=%s" %(self.codetree.basedir)
4656- , "--datadir=%s" %(self.datadir)
4657- , "--tmpdir=%s" %(self.tmpdir)
4658- , "--character-sets-dir=%s" %(self.charsetdir)
4659- , "--lc-messages-dir=%s" %(self.langdir)
4660- , "--ssl-ca=%s" %(os.path.join(self.std_data,'cacert.pem'))
4661- , "--ssl-cert=%s" %(os.path.join(self.std_data,'server-cert.pem'))
4662- , "--ssl-key=%s" %(os.path.join(self.std_data,'server-key.pem'))
4663- , "--port=%d" %(self.master_port)
4664- , "--mysql-protocol.connect-timeout=60"
4665- , "--innodb.data-file-path=ibdata1:20M:autoextend"
4666- , "--sort-buffer-size=256K"
4667- , "--max-heap-table-size=1M"
4668- , "--socket=%s" %(self.socket_file)
4669- , "--pid-file=%s" %(self.pid_file)
4670- , "--default-storage-engine=%s" %(self.default_storage_engine)
4671- , "--server-id=%d" %(self.get_numeric_server_id)
4672- , self.secure_file_string
4673- , self.user_string
4674- ]
4675-
4676- if self.gdb:
4677- server_args.append('--gdb')
4678- return self.system_manager.handle_gdb_reqs(self, server_args)
4679- else:
4680- return "%s %s %s & " % ( self.cmd_prefix
4681- , self.server_path
4682- , " ".join(server_args)
4683- )
4684-
4685-
4686- def get_stop_cmd(self):
4687- """ Return the command that will shut us down """
4688-
4689- return "%s --user=root --port=%d --connect-timeout=5 --silent --password= --shutdown " %(self.mysql_admin, self.master_port)
4690-
4691-
4692- def get_ping_cmd(self):
4693- """Return the command string that will
4694- ping / check if the server is alive
4695-
4696- """
4697-
4698- return '%s --port=%d --user=root -hlocalhost --protocol=tcp -e ""' % (self.mysql_client_path, self.master_port)
4699-
4700- def is_started(self):
4701- """ Determine if the server is up and running -
4702- this may vary from server type to server type
4703-
4704- """
4705-
4706- # We experiment with waiting for a pid file to be created vs. pinging
4707- # This is what test-run.pl does and it helps us pass logging_stats tests
4708- # while not self.ping_server(server, quiet=True) and timer != timeout:
4709-
4710- return self.system_manager.find_path( [self.pid_file]
4711- , required=0)
4712-
4713-
4714
4715=== removed file 'dbqp/lib/server_mgmt/server.py'
4716--- dbqp/lib/server_mgmt/server.py 2012-02-04 01:22:23 +0000
4717+++ dbqp/lib/server_mgmt/server.py 1970-01-01 00:00:00 +0000
4718@@ -1,170 +0,0 @@
4719-#! /usr/bin/env python
4720-# -*- mode: python; indent-tabs-mode: nil; -*-
4721-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
4722-#
4723-# Copyright (C) 2010,2011 Patrick Crews
4724-#
4725-# This program is free software; you can redistribute it and/or modify
4726-# it under the terms of the GNU General Public License as published by
4727-# the Free Software Foundation; either version 2 of the License, or
4728-# (at your option) any later version.
4729-#
4730-# This program is distributed in the hope that it will be useful,
4731-# but WITHOUT ANY WARRANTY; without even the implied warranty of
4732-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4733-# GNU General Public License for more details.
4734-#
4735-# You should have received a copy of the GNU General Public License
4736-# along with this program; if not, write to the Free Software
4737-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
4738-
4739-
4740-""" server.py: generic server object used by the server
4741- manager. This contains the generic methods for all
4742- servers. Specific types (Drizzle, MySQL, etc) should
4743- inherit from this guy
4744-
4745-"""
4746-
4747-# imports
4748-import os
4749-
4750-class Server(object):
4751- """ the server class from which other servers
4752- will inherit - contains generic methods
4753- certain methods will be overridden by more
4754- specific ones
4755-
4756- """
4757-
4758- def __init__(self
4759- , name
4760- , server_manager
4761- , code_tree
4762- , default_storage_engine
4763- , server_options
4764- , requester
4765- , workdir_root):
4766- self.skip_keys = [ 'server_manager'
4767- , 'system_manager'
4768- , 'dirset'
4769- , 'preferred_base_port'
4770- , 'no_secure_file_priv'
4771- , 'secure_file_string'
4772- , 'port_block'
4773- ]
4774- self.debug = server_manager.debug
4775- self.verbose = server_manager.verbose
4776- self.initial_run = 1
4777- self.owner = requester
4778- self.server_options = server_options
4779- self.default_storage_engine = default_storage_engine
4780- self.server_manager = server_manager
4781- # We register with server_manager asap
4782- self.server_manager.log_server(self, requester)
4783-
4784- self.system_manager = self.server_manager.system_manager
4785- self.code_tree = code_tree
4786- self.type = self.code_tree.type
4787- self.valgrind = self.system_manager.valgrind
4788- self.gdb = self.system_manager.gdb
4789- if self.valgrind:
4790- self.valgrind_time_buffer = 10
4791- else:
4792- self.valgrind_time_buffer = 1
4793- self.cmd_prefix = self.system_manager.cmd_prefix
4794- self.logging = self.system_manager.logging
4795- self.no_secure_file_priv = self.server_manager.no_secure_file_priv
4796- self.name = name
4797- self.status = 0 # stopped, 1 = running
4798- self.tried_start = 0
4799- self.failed_test = 0 # was the last test a failure? our state is suspect
4800- self.server_start_timeout = 60 * self.valgrind_time_buffer
4801- self.pid = None
4802- self.ip_address = '127.0.0.1'
4803- self.need_reset = False
4804-
4805- def initialize_databases(self):
4806- """ Call schemawriter to make db.opt files """
4807- databases = [ 'test'
4808- , 'mysql'
4809- ]
4810- for database in databases:
4811- db_path = os.path.join(self.datadir,'local',database,'db.opt')
4812- cmd = "%s %s %s" %(self.schemawriter, database, db_path)
4813- self.system_manager.execute_cmd(cmd)
4814-
4815- def process_server_options(self):
4816- """Consume the list of options we have been passed.
4817- Return a string with them joined
4818-
4819- """
4820-
4821- return " ".join(self.server_options)
4822-
4823- def take_db_snapshot(self):
4824- """ Take a snapshot of our vardir for quick restores """
4825-
4826- self.logging.info("Taking clean db snapshot...")
4827- if os.path.exists(self.snapshot_path):
4828- # We need to remove an existing path as python shutil
4829- # doesn't want an existing target
4830- self.system_manager.remove_dir(self.snapshot_path)
4831- self.system_manager.copy_dir(self.datadir, self.snapshot_path)
4832-
4833- def restore_snapshot(self):
4834- """ Restore from a stored snapshot """
4835-
4836- if not os.path.exists(self.snapshot_path):
4837- self.logging.error("Could not find snapshot: %s" %(self.snapshot_path))
4838- self.system_manager.remove_dir(self.datadir)
4839- self.system_manager.copy_dir(self.snapshot_path, self.datadir)
4840-
4841- def is_started(self):
4842- """ Is the server running? Particulars are server-dependent """
4843-
4844- return "You need to implement is_started"
4845-
4846- def get_start_cmd(self):
4847- """ Return the command the server_manager can use to start me """
4848-
4849- return "You need to implement get_start_cmd"
4850-
4851- def get_stop_cmd(self):
4852- """ Return the command the server_manager can use to stop me """
4853-
4854- return "You need to implement get_stop_cmd"
4855-
4856- def get_ping_cmd(self):
4857- """ Return the command that can be used to 'ping' me
4858- Very similar to is_started, but different
4859-
4860- Determining if a server is still running (ping)
4861- may differ from the method used to determine
4862- server startup
4863-
4864- """
4865-
4866- return "You need to implement get_ping_cmd"
4867-
4868- def cleanup(self):
4869- """ Cleanup - just free ports for now..."""
4870- self.system_manager.port_manager.free_ports(self.port_block)
4871-
4872- def set_server_options(self, server_options):
4873- """ We update our server_options to the new set """
4874- self.server_options = server_options
4875-
4876- def reset(self):
4877- """ Voodoo to reset ourselves """
4878- self.failed_test = 0
4879- self.need_reset = False
4880-
4881- def get_numeric_server_id(self):
4882- """ Return the integer value of server-id
4883- Mainly for mysql / percona, but may be useful elsewhere
4884-
4885- """
4886-
4887- return int(self.name.split(self.server_manager.server_base_name)[1])
4888-
4889
4890=== removed file 'dbqp/lib/server_mgmt/server_management.py'
4891--- dbqp/lib/server_mgmt/server_management.py 2012-02-04 01:22:23 +0000
4892+++ dbqp/lib/server_mgmt/server_management.py 1970-01-01 00:00:00 +0000
4893@@ -1,611 +0,0 @@
4894-#! /usr/bin/env python
4895-# -*- mode: python; indent-tabs-mode: nil; -*-
4896-# vim:expandtab:shiftwidth=2:tabstop=2:smarttab:
4897-#
4898-# Copyright (C) 2010, 2011 Patrick Crews
4899-#
4900-# This program is free software; you can redistribute it and/or modify
4901-# it under the terms of the GNU General Public License as published by
4902-# the Free Software Foundation; either version 2 of the License, or
4903-# (at your option) any later version.
4904-#
4905-# This program is distributed in the hope that it will be useful,
4906-# but WITHOUT ANY WARRANTY; without even the implied warranty of
4907-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
4908-# GNU General Public License for more details.
4909-#
4910-# You should have received a copy of the GNU General Public License
4911-# along with this program; if not, write to the Free Software
4912-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
4913-
4914-"""server_management.py
4915- code for dealing with apportioning servers
4916- to suit the needs of the tests and executors
4917-
4918-"""
4919-# imports
4920-import thread
4921-import time
4922-import os
4923-import subprocess
4924-from ConfigParser import RawConfigParser
4925-
4926-class serverManager:
4927- """ code that handles the server objects
4928- We use this to track, do mass actions, etc
4929- Single point of contact for this business
4930-
4931- """
4932-
4933- def __init__(self, system_manager, variables):
4934- self.skip_keys = [ 'system_manager'
4935- , 'env_manager'
4936- , 'code_manager'
4937- , 'logging'
4938- , 'gdb'
4939- ]
4940- self.debug = variables['debug']
4941- self.verbose = variables['verbose']
4942- self.initial_run = 1
4943- # we try this to shorten things - will see how this works
4944- self.server_base_name = 's'
4945- self.no_secure_file_priv = variables['nosecurefilepriv']
4946- self.system_manager = system_manager
4947- self.code_manager = system_manager.code_manager
4948- self.env_manager = system_manager.env_manager
4949- self.logging = system_manager.logging
4950- self.gdb = self.system_manager.gdb
4951- self.default_storage_engine = variables['defaultengine']
4952- self.default_server_type = variables['defaultservertype']
4953- self.user_server_opts = variables['drizzledoptions']
4954- self.servers = {}
4955-
4956- self.mutex = thread.allocate_lock()
4957- self.timer_increment = .5
4958- self.libeatmydata = variables['libeatmydata']
4959- self.libeatmydata_path = variables['libeatmydatapath']
4960-
4961- self.logging.info("Using default-storage-engine: %s" %(self.default_storage_engine))
4962-
4963- self.logging.debug_class(self)
4964-
4965- def request_servers( self, requester
4966- , workdir, cnf_path
4967- , server_requests
4968- , server_requirements
4969- , working_environ, expect_fail = 0):
4970- """ We produce the server objects / start the server processes
4971- as requested. We report errors and whatnot if we can't
4972- That is, unless we expect the server to not start, then
4973- we just return a value / message.
4974-
4975- server_requirements is a list of lists. Each list
4976- is a set of server options - we create one server
4977- for each set of options requested
4978-
4979- """
4980-
4981- # Make sure our server is in a decent state, if the last test
4982- # failed, then we reset the server
4983- self.check_server_status(requester)
4984-
4985- # Make sure we have the proper number of servers for this requester
4986- self.process_server_count( requester, len(server_requirements)
4987- , workdir, server_requirements)
4988-
4989- # Make sure we are running with the correct options
4990- self.evaluate_existing_servers( requester
4991- , cnf_path
4992- , server_requests
4993- , server_requirements)
4994-
4995- # Fire our servers up
4996- bad_start = self.start_servers( requester, working_environ
4997- , expect_fail)
4998-
4999- # Return them to the requester
5000- return (self.get_server_list(requester), bad_start)
The diff has been truncated for viewing.

Subscribers

People subscribed via source and target branches