Heads up - test runner fix

Hey all,

I have good news and bad news.

Good: our test runners now report failures correctly.

Bad news: our test runners have not been reporting failures since Aug 8 2014.

As of svn r219689, dotest.py will once again report errors/failures via non-zero return value, which will then propagate correctly through test run results as reported by ‘make test’, ‘ninja check-lldb’, etc.

You will almost certainly see test failures show up after r219689 that you were not seeing before. This is the accumulation of all build breaks we essentially made starting with r215256. We will need to fix the errors, not revert r219689. (That would be akin to shooting the messenger).

The lesson here is make sure we don’t break our test runner, and ensure return values from the test runner are preserved on exit. That is a hugely critical facility. I will look into adding a test runner test so we don’t break that again.

This is the accumulation of all build breaks

test breaks, rather…

For example, here is the set of test failures I’m getting on MacOSX 10.10 GM Candidate 3, with the latest Xcode 6.1 beta:

Ran 313 tests.

Failing Tests (6)

FAIL: LLDB (suite) :: TestFormatters.py (Darwin snappy-2.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestCallWithTimeout.py (Darwin snappy-2.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestMemoryHistory.py (Darwin snappy-2.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestReportData.py (Darwin snappy-2.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Darwin snappy-2.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Darwin snappy-2.local 14.0.0 Darwin Kernel Version 14.0.0: Tue Sep 23 23:01:41 PDT 2014; root:xnu-2782.1.97~6/RELEASE_X86_64 x86_64 i386)

The failures I currently get on a stock Ubuntu 14.04 x86_64 Desktop VM running under Fusion 7.0 using TOT as of this morning are these:

Failing Tests (14)

FAIL: LLDB (suite) :: TestGdbRemoteProcessInfo.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestGdbRemoteAttach.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestLldbGdbServer.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestGdbRemoteKill.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestDataFormatterStdVector.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestDataFormatterStdVBool.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestDataFormatterStdMap.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestDataFormatterSynthVal.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestReportData.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestMemoryHistory.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestUniqueTypes.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestAnonymous.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestSharedLib.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

FAIL: LLDB (suite) :: TestSharedLibStrippedSymbols.py (Linux vagrant 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64)

The first 4 or so are mine.