bin/test --load-list does not preserve test order

Bug #989208 reported by Gary Poster
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Brad Crittenden

Bug Description

When trying to duplicate intermittent test failures to see if they are related to the tests that ran before them, we have branches to let us get the tests run on a particular worker (see bug 664171) from the subunit output. Once you have the list, you want to run the tests in that order.

The supported way to run the tests from a list of them is using `bin/test --load-list LIST_NAME` where LIST_NAME is a file of test names separated by newlines. Unfortunately, after the tests are aggregated into layers (expected, and should mirror what happened in the previous test run), they are sorted alphabetically. The only way to disable the alphabetization is to use --shuffle, which randomizes the list. There is no way to maintain the original order specified in the list of test names.

Maintaining the original order is the only way to duplicate the test run exactly. We should be able to do this as one of our diagnostic tools.

This would be a change to our zope.testing fork.

Related branches

Brad Crittenden (bac)
Changed in launchpad:
assignee: nobody → Brad Crittenden (bac)
status: Triaged → In Progress
Revision history for this message
Launchpad QA Bot (lpqabot) wrote :

Fixed in stable r15186 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15186) by a commit, but not testable.

tags: added: qa-untestable
Changed in launchpad:
status: In Progress → Fix Committed
William Grant (wgrant)
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.