[lit] check-all hanging

Hi,

From time to time, I see check-all hang during running of lit tests.

The hang always happens at the > 90% completion stage and I'm pretty
sure all tests have been run and check-all is just waiting for
lit/python to exit. I see a single python processing running, taking
very little CPU time. An strace of that process shows this:

select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32168}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 2000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 4000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 8000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 16000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 32000}) = 0 (Timeout)
futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = 0
futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = -1 EAGAIN (Resourc
e temporarily unavailable)
futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
futex(0x3bcc8c0, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, ffffffff) = -1 EAGAIN (Resourc
e temporarily unavailable)
futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x3bcc8c0, FUTEX_WAKE_PRIVATE, 1) = 1
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {0, 50000}) = 0 (Timeout)

It appears that python is waiting for some I/O or something which never
appears.

Has anyone else seen this before? Any ideas of what is going on or how
to fix it?

                              -David

What you’re seeing is just the fact that lit is waiting on subprocesses (select is waiting on the pipes i suspect).

Anyways, you’ll need to dig into what it is waiting on, and what that process is doing that is stuck to make progress.

I’ve not seen anything like this, but I basically never run check-all these days because LLDB and sanitizer tests are too flaky. =[ I’ve not been able to interest anyone in fixing this either sadly.

Hi David, Chandler,

I see lldb tests hang often, and then I kill the dotest process.

I’d like to stop running check-all too, but I feel it’s important when I modify FileCheck. The flakiness that Chandler mentioned makes it time-consuming to verify test results.

Joel

Might be worth reporting this on the lldb list?

+Fred, +me

For LLDB tests: I believe this got much much better recently. Are you still seeing flaky LLDB tests? Any details you can share?
For sanitizer tests: I’m very much interesting in removing flakiness as well. Any specific tests you see as flaky?

Kuba

All,

Thanks for the replies. Kuba: For LLDB, when were things expected to have improved? It’s possible things improved for me at some point, but this isn’t something I’ve found time to track carefully, and I still see problems.

I ran check-all a couple of times last night at r350238, which I pulled yesterday. Here are the results:

+Fred, +Kostya

-llvm-dev + lldb-dev for the lldv test failures.

All,

Thanks for the replies. Kuba: For LLDB, when were things expected to have improved? It’s possible things improved for me at some point, but this isn’t something I’ve found time to track carefully, and I still see problems.

I ran check-all a couple of times last night at r350238, which I pulled yesterday. Here are the results:

********************
Testing Time: 5043.24s
********************
Unexpected Passing Tests (2):
lldb-Suite :: functionalities/asan/TestMemoryHistory.py
lldb-Suite :: functionalities/asan/TestReportData.py

********************
Failing Tests (54):
Clang :: CXX/modules-ts/basic/[basic.link/p2/module.cpp](http://basic.link/p2/module.cpp)
Clang :: Modules/ExtDebugInfo.cpp
Clang :: Modules/using-directive-redecl.cpp
Clang :: Modules/using-directive.cpp
Clang :: PCH/chain-late-anonymous-namespace.cpp
Clang :: PCH/cxx-namespaces.cpp
Clang :: PCH/namespaces.cpp
LLDB :: ExecControl/StopHook/stop-hook-threads.test
LeakSanitizer-AddressSanitizer-x86_64 :: TestCases/Linux/[use_tls_dynamic.cc](http://use_tls_dynamic.cc)
LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/[use_tls_dynamic.cc](http://use_tls_dynamic.cc)
MemorySanitizer-X86_64 :: dtls_test.c
MemorySanitizer-lld-X86_64 :: dtls_test.c
lldb-Suite :: functionalities/register/register_command/TestRegisters.py
lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py

It’s hard to diagnose dotest failures without the log.

lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestDictionaryResolutionWithDot
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestExtractingUInt64ThroughStructuredData
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionNoDot
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionWithDot
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestInstanceNameResolutionNoDot
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestModuleNameResolutionNoDot
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestObjectAttributes
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestOwnedReferences
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonByteArray
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonBytes
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableCheck
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableInvoke
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryManipulation
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryToStructuredDictionary
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryValueEquality
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonFile
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonInteger
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStr
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStructuredInteger
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListManipulation
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListToStructuredList
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListValueEquality
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonString
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStr
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStructuredString
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList2
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleSize
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleToStructuredList
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleValues
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestResetting
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestTypeNameResolutionNoDot
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAcquisitionSemantics
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreChanged
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreSemantics
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestDiscardSemantics
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestExceptionStateChecking
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestManualRestoreSemantics
lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics

Those unit test failures don’t ring a bell. Anyone’s seen this? Here too, knowing exactly the error message would help.

Expected Passes : 57489
Expected Failures : 276
Unsupported Tests : 1883
Unexpected Passes : 2
Unexpected Failures: 54

14 warning(s) in tests.
FAILED: CMakeFiles/check-all


I immediately ran it again and saw one new unexpected fail:

lldb-Suite :: tools/lldb-mi/syntax/TestMiSyntax.py

Adrian, do we have remaining flakiness in the MI tests? Is this one of the GSoc tests?

and one new unresolved test:

lldb-Suite :: tools/lldb-vscode/breakpoint/TestVSCode_setBreakpoints.py

On the second run but not the first, it hung all night long waiting for TestVSCode_setBreakpoints.py to terminate. I killed dotest.py to get the final results.

We have disabled the VScode tests on Darwin because they very flaky on the bots. +Greg who added those.

Fred

We're not running lldb tests, so something else is going on. I'll dig
into it.

                         -David

"Joel E. Denny" <jdenny.ornl@gmail.com> writes:

Chandler Carruth via llvm-dev <llvm-dev@lists.llvm.org> writes:

What you're seeing is just the fact that lit is waiting on
subprocesses (select is waiting on the pipes i suspect).

Right. Some digging revealed that it is waiting on
getline_nohang.cc.tmp, a tsan test.

I see that this test has been disabled for NetBSD, due to it sometimes
failing. I'm seeing the same on Linux.

How can we stabilize the sanitizer tests so that check-all can work
reliably? If some sanitizer tests are so flaky, I should think they
should be marked UNSUPPORTED. Who has the authority to make those
determinations?

                        -David

-llvm-dev + lldb-dev for the lldv test failures.

Expected Passes : 57489
Expected Failures : 276
Unsupported Tests : 1883
Unexpected Passes : 2
Unexpected Failures: 54

14 warning(s) in tests.
FAILED: CMakeFiles/check-all


I immediately ran it again and saw one new unexpected fail:

lldb-Suite :: tools/lldb-mi/syntax/TestMiSyntax.py

Adrian, do we have remaining flakiness in the MI tests? Is this one of the GSoc tests?

That particular test still uses pexpect and needs to be rewritten as a LIT-based test; so, yes.

– adrian

Chandler Carruth via llvm-dev <llvm-dev@lists.llvm.org> writes:

What you're seeing is just the fact that lit is waiting on
subprocesses (select is waiting on the pipes i suspect).

Right. Some digging revealed that it is waiting on
getline_nohang.cc.tmp, a tsan test.

I see that this test has been disabled for NetBSD, due to it sometimes
failing. I'm seeing the same on Linux.

How can we stabilize the sanitizer tests so that check-all can work
reliably? If some sanitizer tests are so flaky, I should think they
should be marked UNSUPPORTED. Who has the authority to make those
determinations?

Dmitry Vyukov does. CC'ing him.

Kuba

TestMiSyntax was the only test failing for me with /usr/lib/debug debuginfos
installed on Linux and Joel may be running a Linux platform according to some
of his mails.

I have now checked in a fix which may remove this false FAIL:
https://reviews.llvm.org/D55859 symbols.enable-external-lookup=false on all hosts (not just OSX)

Jan

Are there any special repro instructions? I am running all tsan tests
periodically on linux and none of them flakes.

(My last reply to this was rejected by the list because I wasn’t subscribed. Trying again.)

I have no experience debugging lldb. Here’s the lit output for the last fail (now at r350377), but let me know if you want something more:


FAIL: lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py (59083 of 59736)
******************** TEST 'lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py' FAILED ********************
lldb version 8.0.0
LLDB library dir: /home/jdenny/ornl/llvm-mono-git-build/bin
LLDB import library dir: /home/jdenny/ornl/llvm-mono-git-build/bin
Libc++ tests will not be run because: Unable to find libc++ installation
Skipping following debug info categories: ['dsym', 'gmodules']

Session logs for test failures/errors/unexpected successes will go into directory '/home/jdenny/ornl/llvm-mono-git-build/lldb-test-traces'
Command invoked: /home/jdenny/ornl/llvm-mono-git/lldb/test/dotest.py -q --arch=x86_64 -s /home/jdenny/ornl/llvm-mono-git-build/lldb-test-traces --build-dir /home/jdenny/ornl/llvm-mono-git-build/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /home/jdenny/ornl/llvm-mono-git-build/./bin/lldb --dsymutil /home/jdenny/ornl/llvm-mono-git-build/./bin/dsymutil --filecheck /home/jdenny/ornl/llvm-mono-git-build/./bin/FileCheck -C /home/jdenny/ornl/llvm-mono-git-build/./bin/clang --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy /home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server -p TestGdbRemoteRegisterState.py
UNSUPPORTED: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_no_suffix_debugserver (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState) (debugserver tests)
FAIL: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_no_suffix_llgs (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState)
lldb-server exiting...
UNSUPPORTED: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_with_suffix_debugserver (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState) (debugserver tests)
FAIL: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_with_suffix_llgs (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState)
lldb-server exiting...

-llvm-dev + lldb-dev for the lldv test failures.

All,

Thanks for the replies. Kuba: For LLDB, when were things expected to have improved? It's possible things improved for me at some point, but this isn't something I've found time to track carefully, and I still see problems.

I ran check-all a couple of times last night at r350238, which I pulled yesterday. Here are the results:

********************
Testing Time: 5043.24s
********************
Unexpected Passing Tests (2):
    lldb-Suite :: functionalities/asan/TestMemoryHistory.py
    lldb-Suite :: functionalities/asan/TestReportData.py

********************
Failing Tests (54):
    Clang :: CXX/modules-ts/basic/basic.link/p2/module.cpp
    Clang :: Modules/ExtDebugInfo.cpp
    Clang :: Modules/using-directive-redecl.cpp
    Clang :: Modules/using-directive.cpp
    Clang :: PCH/chain-late-anonymous-namespace.cpp
    Clang :: PCH/cxx-namespaces.cpp
    Clang :: PCH/namespaces.cpp
    LLDB :: ExecControl/StopHook/stop-hook-threads.test
    LeakSanitizer-AddressSanitizer-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
    LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
    MemorySanitizer-X86_64 :: dtls_test.c
    MemorySanitizer-lld-X86_64 :: dtls_test.c
    lldb-Suite :: functionalities/register/register_command/TestRegisters.py
    lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py

It’s hard to diagnose dotest failures without the log.

(My last reply to this was rejected by the list because I wasn’t subscribed. Trying again.)

I have no experience debugging lldb. Here’s the lit output for the last fail (now at r350377), but let me know if you want something more:

FAIL: lldb\-Suite :: tools/lldb\-server/TestGdbRemoteRegisterState\.py \(59083 of 59736\)
\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* TEST &#39;lldb\-Suite :: tools/lldb\-server/TestGdbRemoteRegisterState\.py&#39; FAILED \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
lldb version 8\.0\.0
LLDB library dir: /home/jdenny/ornl/llvm\-mono\-git\-build/bin
LLDB import library dir: /home/jdenny/ornl/llvm\-mono\-git\-build/bin
Libc\+\+ tests will not be run because: Unable to find libc\+\+ installation
Skipping following debug info categories: \[&#39;dsym&#39;, &#39;gmodules&#39;\]

Session logs for test failures/errors/unexpected successes will go into directory &#39;/home/jdenny/ornl/llvm\-mono\-git\-build/lldb\-test\-traces&#39;
Command invoked: /home/jdenny/ornl/llvm\-mono\-git/lldb/test/dotest\.py \-q \-\-arch=x86\_64 \-s /home/jdenny/ornl/llvm\-mono\-git\-build/lldb\-test\-traces \-\-build\-dir /home/jdenny/ornl/llvm\-mono\-git\-build/lldb\-test\-build\.noindex \-S nm \-u CXXFLAGS \-u CFLAGS \-\-executable /home/jdenny/ornl/llvm\-mono\-git\-build/\./bin/lldb \-\-dsymutil /home/jdenny/ornl/llvm\-mono\-git\-build/\./bin/dsymutil \-\-filecheck /home/jdenny/ornl/llvm\-mono\-git\-build/\./bin/FileCheck \-C /home/jdenny/ornl/llvm\-mono\-git\-build/\./bin/clang \-\-env ARCHIVER=/usr/bin/ar \-\-env OBJCOPY=/usr/bin/objcopy /home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server \-p TestGdbRemoteRegisterState\.py
UNSUPPORTED: LLDB \(/home/jdenny/ornl/llvm\-mono\-git\-build/bin/clang\-8\-x86\_64\) :: test\_grp\_register\_save\_restore\_works\_no\_suffix\_debugserver \(TestGdbRemoteRegisterState\.TestGdbRemoteRegisterState\) \(debugserver tests\) 
FAIL: LLDB \(/home/jdenny/ornl/llvm\-mono\-git\-build/bin/clang\-8\-x86\_64\) :: test\_grp\_register\_save\_restore\_works\_no\_suffix\_llgs \(TestGdbRemoteRegisterState\.TestGdbRemoteRegisterState\)
lldb\-server exiting\.\.\.
UNSUPPORTED: LLDB \(/home/jdenny/ornl/llvm\-mono\-git\-build/bin/clang\-8\-x86\_64\) :: test\_grp\_register\_save\_restore\_works\_with\_suffix\_debugserver \(TestGdbRemoteRegisterState\.TestGdbRemoteRegisterState\) \(debugserver tests\) 
FAIL: LLDB \(/home/jdenny/ornl/llvm\-mono\-git\-build/bin/clang\-8\-x86\_64\) :: test\_grp\_register\_save\_restore\_works\_with\_suffix\_llgs \(TestGdbRemoteRegisterState\.TestGdbRemoteRegisterState\)
lldb\-server exiting\.\.\.
======================================================================
FAIL: test\_grp\_register\_save\_restore\_works\_no\_suffix\_llgs \(TestGdbRemoteRegisterState\.TestGdbRemoteRegisterState\)
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Traceback \(most recent call last\):
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/decorators\.py&quot;, line 144, in wrapper
&nbsp;&nbsp;&nbsp;&nbsp;func\(\*args, \*\*kwargs\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/TestGdbRemoteRegisterState\.py&quot;, line 137, in test\_grp\_register\_save\_restore\_works\_no\_suffix\_llgs
&nbsp;&nbsp;&nbsp;&nbsp;self\.grp\_register\_save\_restore\_works\(USE\_THREAD\_SUFFIX\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/TestGdbRemoteRegisterState\.py&quot;, line 97, in grp\_register\_save\_restore\_works
&nbsp;&nbsp;&nbsp;&nbsp;context = self\.expect\_gdbremote\_sequence\(\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/gdbremote\_testcase\.py&quot;, line 713, in expect\_gdbremote\_sequence
&nbsp;&nbsp;&nbsp;&nbsp;self\.logger\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 261, in expect\_lldb\_gdbserver\_replay
&nbsp;&nbsp;&nbsp;&nbsp;asserter, content, context=context\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 564, in assert\_match
&nbsp;&nbsp;&nbsp;&nbsp;self\.\_assert\_exact\_payload\_match\(asserter, actual\_packet\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 518, in \_assert\_exact\_payload\_match
&nbsp;&nbsp;&nbsp;&nbsp;assert\_packets\_equal\(asserter, actual\_packet, self\.exact\_payload\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 159, in assert\_packets\_equal
&nbsp;&nbsp;&nbsp;&nbsp;asserter\.assertEqual\(actual\_stripped, expected\_stripped\)
AssertionError: &#39;$E77&#39; \!= &#39;$OK&#39;
Config=x86\_64\-/home/jdenny/ornl/llvm\-mono\-git\-build/bin/clang\-8
======================================================================
FAIL: test\_grp\_register\_save\_restore\_works\_with\_suffix\_llgs \(TestGdbRemoteRegisterState\.TestGdbRemoteRegisterState\)
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Traceback \(most recent call last\):
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/decorators\.py&quot;, line 144, in wrapper
&nbsp;&nbsp;&nbsp;&nbsp;func\(\*args, \*\*kwargs\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/TestGdbRemoteRegisterState\.py&quot;, line 121, in test\_grp\_register\_save\_restore\_works\_with\_suffix\_llgs
&nbsp;&nbsp;&nbsp;&nbsp;self\.grp\_register\_save\_restore\_works\(USE\_THREAD\_SUFFIX\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/TestGdbRemoteRegisterState\.py&quot;, line 97, in grp\_register\_save\_restore\_works
&nbsp;&nbsp;&nbsp;&nbsp;context = self\.expect\_gdbremote\_sequence\(\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/gdbremote\_testcase\.py&quot;, line 713, in expect\_gdbremote\_sequence
&nbsp;&nbsp;&nbsp;&nbsp;self\.logger\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 261, in expect\_lldb\_gdbserver\_replay
&nbsp;&nbsp;&nbsp;&nbsp;asserter, content, context=context\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 564, in assert\_match
&nbsp;&nbsp;&nbsp;&nbsp;self\.\_assert\_exact\_payload\_match\(asserter, actual\_packet\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 518, in \_assert\_exact\_payload\_match
&nbsp;&nbsp;&nbsp;&nbsp;assert\_packets\_equal\(asserter, actual\_packet, self\.exact\_payload\)
&nbsp;&nbsp;File &quot;/home/jdenny/ornl/llvm\-mono\-git/lldb/packages/Python/lldbsuite/test/tools/lldb\-server/lldbgdbserverutils\.py&quot;, line 159, in assert\_packets\_equal
&nbsp;&nbsp;&nbsp;&nbsp;asserter\.assertEqual\(actual\_stripped, expected\_stripped\)
AssertionError: &#39;$E77&#39; \!= &#39;$OK&#39;
Config=x86\_64\-/home/jdenny/ornl/llvm\-mono\-git\-build/bin/clang\-8
\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-
Ran 4 tests in 22\.052s

I’m not familiar with the lldb-server tests. Pavel might know what Error 77 is?

RESULT: FAILED (0 passes, 2 failures, 0 errors, 2 skipped, 0 expected failures, 0 unexpected successes)

********************


>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestDictionaryResolutionWithDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestExtractingUInt64ThroughStructuredData
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionWithDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestInstanceNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestModuleNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestObjectAttributes
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestOwnedReferences
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonByteArray
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonBytes
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableCheck
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableInvoke
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryManipulation
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryToStructuredDictionary
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryValueEquality
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonFile
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonInteger
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStr
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStructuredInteger
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListManipulation
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListToStructuredList
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListValueEquality
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonString
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStr
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStructuredString
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList2
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleSize
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleToStructuredList
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleValues
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestResetting
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestTypeNameResolutionNoDot
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAcquisitionSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreChanged
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestDiscardSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestExceptionStateChecking
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestManualRestoreSemantics
>     lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics

Those unit test failures don’t ring a bell. Anyone’s seen this? Here too, knowing exactly the error message would help.

Here's the lit output for the last one:

FAIL: lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics (59324 of 59736)
******************** TEST 'lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics' FAILED ********************
Note: Google Test filter = PythonExceptionStateTest.TestResetSemantics
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from PythonExceptionStateTest
[ RUN ] PythonExceptionStateTest.TestResetSemantics
ScriptInterpreterPythonTests: /home/jdenny/ornl/llvm-mono-git/llvm/include/llvm/Support/CommandLine.h:800: void llvm::cl::parser<llvm::ScheduleDAGInstrs *(*)(llvm::MachineSchedContext *)>::addLiteralOption(llvm::StringRef, const DT &, llvm::StringRef) [DataType = llvm::ScheduleDAGInstrs *(*)(llvm::MachineSchedContext *), DT = llvm::ScheduleDAGInstrs *(*)(llvm::MachineSchedContext *)]: Assertion `findOption(Name) == Values.size() && "Option already exists!"' failed.

That’s interesting. It seems like some clots are registered twice. As he backtrace shows that you have a libLLVMSupport.so, I could imagine this happening if the unittest was linking with both a static version and the dynamic version of the LLVM libraries, but that’s just a theory. I don’t know how good the build system is at dealing with a libraries build of LLVM, I wouldn’t be surprised if this is untested and thus broken.

Fred

Dmitry Vyukov <dvyukov@google.com> writes:

Are there any special repro instructions? I am running all tsan tests
periodically on linux and none of them flakes.

I don't think I'm doing anything especially interesting. I wonder if
lit parallelism has anything to do with it. I tend to run quite wide
(32 or more).

I'm on SLES 12.2, kernel 4.4.21-69-default, x86_64 in case it matters.
I see this test hang pretty frequently.

                            -David

Hi David,

The test is specifically a regression test for a deadlock:

// Make sure TSan doesn't deadlock on a file stream lock at program shutdown.
// See https://github.com/google/sanitizers/issues/454

So I wonder if it's not completely fixed.
I am sure it does not reproduce on my machine:

$ clang++ getline_nohang.cc -fsanitize=thread -O1 -g
$ stress ./a.out
192 runs so far, 0 failures
...
17137 runs so far, 0 failures
17377 runs so far, 0 failures

Could you please attach to the hanged process with gdb and do
backtrace of all threads?

-llvm-dev + lldb-dev for the lldv test failures.

All,

Thanks for the replies. Kuba: For LLDB, when were things expected to have improved? It’s possible things improved for me at some point, but this isn’t something I’ve found time to track carefully, and I still see problems.

I ran check-all a couple of times last night at r350238, which I pulled yesterday. Here are the results:

********************
Testing Time: 5043.24s
********************
Unexpected Passing Tests (2):
lldb-Suite :: functionalities/asan/TestMemoryHistory.py
lldb-Suite :: functionalities/asan/TestReportData.py

********************
Failing Tests (54):
Clang :: CXX/modules-ts/basic/basic.link/p2/module.cpp
Clang :: Modules/ExtDebugInfo.cpp
Clang :: Modules/using-directive-redecl.cpp
Clang :: Modules/using-directive.cpp
Clang :: PCH/chain-late-anonymous-namespace.cpp
Clang :: PCH/cxx-namespaces.cpp
Clang :: PCH/namespaces.cpp
LLDB :: ExecControl/StopHook/stop-hook-threads.test
LeakSanitizer-AddressSanitizer-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
LeakSanitizer-Standalone-x86_64 :: TestCases/Linux/use_tls_dynamic.cc
MemorySanitizer-X86_64 :: dtls_test.c
MemorySanitizer-lld-X86_64 :: dtls_test.c
lldb-Suite :: functionalities/register/register_command/TestRegisters.py
lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py

It’s hard to diagnose dotest failures without the log.

(My last reply to this was rejected by the list because I wasn’t subscribed. Trying again.)

I have no experience debugging lldb. Here’s the lit output for the last fail (now at r350377), but let me know if you want something more:

FAIL: lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py (59083 of 59736)
******************** TEST 'lldb-Suite :: tools/lldb-server/TestGdbRemoteRegisterState.py' FAILED ********************
lldb version 8.0.0
LLDB library dir: /home/jdenny/ornl/llvm-mono-git-build/bin
LLDB import library dir: /home/jdenny/ornl/llvm-mono-git-build/bin
Libc++ tests will not be run because: Unable to find libc++ installation
Skipping following debug info categories: ['dsym', 'gmodules']

Session logs for test failures/errors/unexpected successes will go into directory '/home/jdenny/ornl/llvm-mono-git-build/lldb-test-traces'
Command invoked: /home/jdenny/ornl/llvm-mono-git/lldb/test/dotest.py -q --arch=x86_64 -s /home/jdenny/ornl/llvm-mono-git-build/lldb-test-traces --build-dir /home/jdenny/ornl/llvm-mono-git-build/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /home/jdenny/ornl/llvm-mono-git-build/./bin/lldb --dsymutil /home/jdenny/ornl/llvm-mono-git-build/./bin/dsymutil --filecheck /home/jdenny/ornl/llvm-mono-git-build/./bin/FileCheck -C /home/jdenny/ornl/llvm-mono-git-build/./bin/clang --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy /home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server -p TestGdbRemoteRegisterState.py
UNSUPPORTED: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_no_suffix_debugserver (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState) (debugserver tests)
FAIL: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_no_suffix_llgs (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState)
lldb-server exiting...
UNSUPPORTED: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_with_suffix_debugserver (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState) (debugserver tests)
FAIL: LLDB (/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8-x86_64) :: test_grp_register_save_restore_works_with_suffix_llgs (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState)
lldb-server exiting...
======================================================================
FAIL: test_grp_register_save_restore_works_no_suffix_llgs (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/decorators.py", line 144, in wrapper
func(*args, **kwargs)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/TestGdbRemoteRegisterState.py", line 137, in test_grp_register_save_restore_works_no_suffix_llgs
self.grp_register_save_restore_works(USE_THREAD_SUFFIX)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/TestGdbRemoteRegisterState.py", line 97, in grp_register_save_restore_works
context = self.expect_gdbremote_sequence()
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/gdbremote_testcase.py", line 713, in expect_gdbremote_sequence
self.logger)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 261, in expect_lldb_gdbserver_replay
asserter, content, context=context)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 564, in assert_match
self._assert_exact_payload_match(asserter, actual_packet)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 518, in _assert_exact_payload_match
assert_packets_equal(asserter, actual_packet, self.exact_payload)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 159, in assert_packets_equal
asserter.assertEqual(actual_stripped, expected_stripped)
AssertionError: '$E77' != '$OK'
Config=x86_64-/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8
======================================================================
FAIL: test_grp_register_save_restore_works_with_suffix_llgs (TestGdbRemoteRegisterState.TestGdbRemoteRegisterState)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/decorators.py", line 144, in wrapper
func(*args, **kwargs)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/TestGdbRemoteRegisterState.py", line 121, in test_grp_register_save_restore_works_with_suffix_llgs
self.grp_register_save_restore_works(USE_THREAD_SUFFIX)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/TestGdbRemoteRegisterState.py", line 97, in grp_register_save_restore_works
context = self.expect_gdbremote_sequence()
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/gdbremote_testcase.py", line 713, in expect_gdbremote_sequence
self.logger)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 261, in expect_lldb_gdbserver_replay
asserter, content, context=context)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 564, in assert_match
self._assert_exact_payload_match(asserter, actual_packet)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 518, in _assert_exact_payload_match
assert_packets_equal(asserter, actual_packet, self.exact_payload)
File "/home/jdenny/ornl/llvm-mono-git/lldb/packages/Python/lldbsuite/test/tools/lldb-server/lldbgdbserverutils.py", line 159, in assert_packets_equal
asserter.assertEqual(actual_stripped, expected_stripped)
AssertionError: '$E77' != '$OK'
Config=x86_64-/home/jdenny/ornl/llvm-mono-git-build/bin/clang-8
----------------------------------------------------------------------
Ran 4 tests in 22.052s

I’m not familiar with the lldb-server tests. Pavel might know what Error 77 is?

RESULT: FAILED (0 passes, 2 failures, 0 errors, 2 skipped, 0 expected failures, 0 unexpected successes)




> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestDictionaryResolutionWithDot
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestExtractingUInt64ThroughStructuredData
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionNoDot
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestGlobalNameResolutionWithDot
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestInstanceNameResolutionNoDot
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestModuleNameResolutionNoDot
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestObjectAttributes
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestOwnedReferences
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonByteArray
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonBytes
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableCheck
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonCallableInvoke
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryManipulation
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryToStructuredDictionary
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonDictionaryValueEquality
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonFile
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonInteger
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStr
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonIntegerToStructuredInteger
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListManipulation
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListToStructuredList
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonListValueEquality
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonString
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStr
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonStringToStructuredString
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleInitializerList2
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleSize
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleToStructuredList
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestPythonTupleValues
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestResetting
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestTypeNameResolutionNoDot
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAcquisitionSemantics
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreChanged
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestAutoRestoreSemantics
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestDiscardSemantics
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestExceptionStateChecking
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestManualRestoreSemantics
> lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics

Those unit test failures don’t ring a bell. Anyone’s seen this? Here too, knowing exactly the error message would help.

Here's the lit output for the last one:

FAIL: lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics (59324 of 59736)
******************** TEST ‘lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonExceptionStateTest.TestResetSemantics’ FAILED ********************
Note: Google Test filter = PythonExceptionStateTest.TestResetSemantics
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from PythonExceptionStateTest
[ RUN ] PythonExceptionStateTest.TestResetSemantics
ScriptInterpreterPythonTests: /home/jdenny/ornl/llvm-mono-git/llvm/include/llvm/Support/CommandLine.h:800: void llvm::cl::parser<llvm::ScheduleDAGInstrs ()(llvm::MachineSchedContext *)>::addLiteralOption(llvm::StringRef, const DT &, llvm::StringRef) [DataType = llvm::ScheduleDAGInstrs ()(llvm::MachineSchedContext *), DT = llvm::ScheduleDAGInstrs ()(llvm::MachineSchedContext *)]: Assertion `findOption(Name) == Values.size() && “Option already exists!”’ failed.

That’s interesting. It seems like some clots are registered twice. As he backtrace shows that you have a libLLVMSupport.so, I could imagine this happening if the unittest was linking with both a static version and the dynamic version of the LLVM libraries, but that’s just a theory. I don’t know how good the build system is at dealing with a libraries build of LLVM, I wouldn’t be surprised if this is untested and thus broken.

I rebuilt without -DBUILD_SHARED_LIBS=true, and all the previous failing clang and lldb-Unit tests then passed (other test results did not change). My build size jumped from 33G to 136G. I didn’t pay attention to the build time this time, but my past experience is that incremental builds can be many times slower without -DBUILD_SHARED_LIBS=true. This does not seem like a good solution for me.

Joel

I think the number of people using BUILD_SHARED_LIBS=True is
relatively limited, and there's no bot testing that configuration.
Stefan is the resident build expert, so he might be able to comment.