Test framework re-compiles for each test case

Bug #589000 reported by Matt Giuca
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mars
Fix Released
High
Matt Giuca

Bug Description

The test framework is rather slow (especially when heavy analysis is performed) because it re-compiles each module from scratch for each test case. This results in N^2 performance for a given test module (if there is roughly one test case per function, the framework will run N tests, and for each test, will fully compile ~N functions). The result is that it is much faster to have each test case in its own module than lump them all together (which is undesirable).

The cause is that each runtime test and unit test calls `mars -c` which compiles the program, then runs a single command. The compilation testing calls `marsc`, and an additional call is made to `mars -c :ta` to get a list of all functions for unit testing. These can all be combined into a single compilation by using pipes.

Replace the initial call to `marsc` (the compilation testing) with a call to `mars -i` which performs compilation, and starts the interactive mode. This should check for errors, and if there are none, result in a Popen-like object which retains a persistent connection with the Mars prompt. All subsequent calls to run_mars should be replaced to a method call on this object, which enters a command into the prompt and checks its feedback without terminating the process. This should not result in interference between test cases (unless they read and write interactive variables, which they should not).

Tags: testing

Related branches

Revision history for this message
Matt Giuca (mgiuca) wrote :

Woohoo! The current test suite takes 25 seconds to run on my laptop. The fast-testing branch, 2 seconds! I expect an even greater speedup on the (private) heavy-analysis branches.

Revision history for this message
Matt Giuca (mgiuca) wrote :

Another (private) branch, with much heavier analysis and many more tests, takes 8 minutes 19 to run. With the fast-testing branch merged, 22 seconds, a 22-fold speedup. Neat! :)

Revision history for this message
Matt Giuca (mgiuca) wrote :

Fixed in trunk r1082.

Changed in mars:
milestone: none → 0.3
status: Triaged → Fix Committed
Matt Giuca (mgiuca)
Changed in mars:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.