[Bug 27199] New: test infra: an exceptional exit outside of any test method should be cleared on rerun with no exceptional exit

Bug ID 27199
Summary test infra: an exceptional exit outside of any test method should be cleared on rerun with no exceptional exit
Product lldb
Version unspecified
Hardware PC
OS All
Status NEW
Severity normal
Priority P
Component All Bugs
Assignee lldb-dev@lists.llvm.org
Reporter todd.fiala@gmail.com
CC llvm-bugs@lists.llvm.org
Classification Unclassified

Currently all test issues (timeout, exceptional exit, failure, unexpected
success, etc.) are tracked to the method they are running within.  When no test
method was running, the charging of process level events like exceptional exits
and timeouts really don't have enough info to assign the issue to a test
method, and thus don't have a great way to figure out if the rerun clears that
issue.  And right now the rerun won't clear events that get tagged to the
process level.

However, there is one case that seems to me to be crystal clear: if the running
of the test methods in a file causes an exceptional exit, and we then rerun
that file, if there is no exceptional exit on the rerun, we should be good.

It seems like if we're rerunning a file's set of test methods, we could clear
file-level issues.  If they happen again, we'll pick them up again and report
them.  If they don't happen, then the rerun addressed whatever was initially
causing the issue.

Here's one recent example on the OS X Green Dragon llvm.org builder:

[http://lab.llvm.org:8080/green/job/lldb_build_test/17392/console](http://lab.llvm.org:8080/green/job/lldb_build_test/17392/console)

=== [SNIP] ===

Rerunning the following files:
  functionalities/thread/state/TestThreadStates.py
rerun will use the 'threading' test runner strategy
Testing: 1 test suites, 1 thread

0 out of 1 test suites processed - 
1 out of 1 test suites processed - TestThreadStates.py
Test rerun complete

Todd Fiala changed bug 27199