Running check-lldb

Scott_Smith · April 18, 2017, 8:24pm

I’m trying to make sure some of my changes don’t break lldb tests, but I’m having trouble getting a clean run even with a plain checkout. I’ve tried the latest head of master, as well as release_40. I’m running Ubuntu 16.04/amd64. I built with:

cmake …/llvm -G Ninja -DCMAKE_BUILD_TYPE=Debug

ninja lldb

ninja check-lldb

Compiler is gcc-5.4, though I’ve also tried with clang-4.0.

Am I missing something obvious? Is there a docker image / vm image / known good environments that I can use to reproduce a clean test run (on something Linux-y - sorry, I don’t have a Mac)?

Pavel_Labath · April 19, 2017, 10:59am

It looks like we are triggering an assert in llvm on a debug build. I’ll try to track this down ASAP.

Scott_Smith · April 19, 2017, 1:28pm

Yeah I found the buildbot instance for lldb on Ubuntu 14.04, but it looks like it is only running release builds. Is that on purpose?

Pavel_Labath · April 19, 2017, 2:37pm

It is on purpose, although whether that purpose is worthwhile is debatable…

We chose to run release builds there so to align the bots closer to the binaries we release. Unfortunately, it does mean we run into situations like these…

In any case, I have now a patch up for fixing one of the crashers. The main one (assert during relocation processing) seems to be caused by a recent change in llvm. I am working towards identifying the cause, but that may take a while.

Then we can hopefully have a look at failures on your machine.

jingham · April 19, 2017, 6:03pm

It is on purpose, although whether that purpose is worthwhile is debatable...

We chose to run release builds there so to align the bots closer to the binaries we release. Unfortunately, it does mean we run into situations like these...

We should be running at least some bots with asserts enabled to catch this sort of issue.

The reason we release lldb with asserts off is that at least historically llvm/clang would assert too aggressively, and crash the debugger in many cases where it would have survived just fine. If you crash a compile session that was at best going to present a compile error, that's not so bad. But having the debugger crash mid-session because an expression went a little sour is not acceptable...

Maybe that situation has changed over time, and we can re-assess this decision? Though the below indicates maybe not???

Jim

Scott_Smith · April 19, 2017, 6:15pm

A combination of:

Updating to a known good release according to buildbot
using Ubuntu 14.04
compiling release using clang-4.0
using the dotest command line that buildbot uses
specifying gcc-4.8 instead of the locally compiled clang

has most of the tests passing, with a handful of unexpected successes:

UNEXPECTED SUCCESS: TestRegisterVariables.RegisterVariableTestCase.test_and_run_command_dwarf (lang/c/register_variables/TestRegisterVariables.py)
UNEXPECTED SUCCESS: TestRegisterVariables.RegisterVariableTestCase.test_and_run_command_dwo (lang/c/register_variables/TestRegisterVariables.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwarf (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwo (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestThreadStates.ThreadStateTestCase.test_process_interrupt_dwarf (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: TestThreadStates.ThreadStateTestCase.test_process_interrupt_dwo (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwarf (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwo (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwarf (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwo (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwarf (lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwo (lang/cpp/printf/TestPrintf.py)

This looks different than another user’s issue: http://lists.llvm.org/pipermail/lldb-dev/2016-February/009504.html

I also tried gcc-4.9.4 (via the ubuntu-toolchain-r ppa) and got a different set of problems:

FAIL: TestNamespaceDefinitions.NamespaceDefinitionsTestCase.test_expr_dwarf (lang/cpp/namespace_definitions/TestNamespaceDefinitions.py)
FAIL: TestNamespaceDefinitions.NamespaceDefinitionsTestCase.test_expr_dwo (lang/cpp/namespace_definitions/TestNamespaceDefinitions.py)
FAIL: TestTopLevelExprs.TopLevelExpressionsTestCase.test_top_level_expressions_dwarf (expression_command/top-level/TestTopLevelExprs.py)
FAIL: TestTopLevelExprs.TopLevelExpressionsTestCase.test_top_level_expressions_dwo (expression_command/top-level/TestTopLevelExprs.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwarf (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwo (functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestThreadStates.ThreadStateTestCase.test_process_interrupt_dwarf (functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwarf (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwo (functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwarf (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwo (api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwarf (lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwo (lang/cpp/printf/TestPrintf.py)

Well, at least the list is consistent, which gives me a base to start testing race conditions

Pavel_Labath · April 20, 2017, 1:47pm

A combination of:
1. Updating to a known good release according to buildbot
2. using Ubuntu 14.04
3. compiling release using clang-4.0

I'd hope that the compiler used to build lldb does not matter. If you see
any differences due to this factor, please let me know.

4. using the dotest command line that buildbot uses

The exact command line the buildbot uses is not important.. The only
important distinction from the check-lldb target is the compiler used. By
default it uses the host compiler used to build lldb. As no-one builds
tests using clang-4.0 it's quite possible that some things may be broken
(or just not properly annotated).

5. specifying gcc-4.8 instead of the locally compiled clang

has most of the tests passing, with a handful of unexpected successes:

UNEXPECTED SUCCESS: TestRegisterVariables.RegisterVariableTestCase.test_and_run_command_dwarf
(lang/c/register_variables/TestRegisterVariables.py)
UNEXPECTED SUCCESS: TestRegisterVariables.RegisterVariableTestCase.test_and_run_command_dwo
(lang/c/register_variables/TestRegisterVariables.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwarf
(functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwo
(functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestThreadStates.ThreadStateTestCase.test_process_interrupt_dwarf
(functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: TestThreadStates.ThreadStateTestCase.test_process_interrupt_dwo
(functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwarf
(functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwo
(functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwarf
(api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwo
(api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwarf
(lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwo
(lang/cpp/printf/TestPrintf.py)

The unexpected successes are expected, unfortunately. What happens here
is that the tests are flaky and they fail like 1% of the time, so they are
marked as xfail.

This looks different than another user's issue: http://lists.llvm.org/
pipermail/lldb-dev/2016-February/009504.html

I also tried gcc-4.9.4 (via the ubuntu-toolchain-r ppa) and got a
different set of problems:

FAIL: TestNamespaceDefinitions.NamespaceDefinitionsTestCase.test_expr_dwarf
(lang/cpp/namespace_definitions/TestNamespaceDefinitions.py)
FAIL: TestNamespaceDefinitions.NamespaceDefinitionsTestCase.test_expr_dwo
(lang/cpp/namespace_definitions/TestNamespaceDefinitions.py)
FAIL: TestTopLevelExprs.TopLevelExpressionsTestCase.
test_top_level_expressions_dwarf (expression_command/top-level/
TestTopLevelExprs.py)
FAIL: TestTopLevelExprs.TopLevelExpressionsTestCase.test_top_level_expressions_dwo
(expression_command/top-level/TestTopLevelExprs.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwarf
(functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestExitDuringBreak.ExitDuringBreakpointTestCase.test_dwo
(functionalities/thread/exit_during_break/TestExitDuringBreak.py)
UNEXPECTED SUCCESS: TestThreadStates.ThreadStateTestCase.test_process_interrupt_dwarf
(functionalities/thread/state/TestThreadStates.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwarf
(functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestRaise.RaiseTestCase.test_restart_bug_dwo
(functionalities/signal/raise/TestRaise.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwarf
(api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: TestMultithreaded.SBBreakpointCallbackCase.test_sb_api_listener_resume_dwo
(api/multithreaded/TestMultithreaded.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwarf
(lang/cpp/printf/TestPrintf.py)
UNEXPECTED SUCCESS: lldbsuite.test.lldbtest.TestPrintf.test_with_dwo
(lang/cpp/printf/TestPrintf.py)

Well, at least the list is consistent, which gives me a base to start
testing race conditions

Interesting, the bot uses a locally-built gcc-4.9.2 and those tests seem to
work there... I'll see if I can find the time to check that out.

Could you please also share the list of failures when using the top-of-tree
clang. I've tried that configuration yesterday on my home machine (x86_64
gentoo) and it had only a couple of failures -- most of them should be
fixed now, only a single failure in TestExprs2 should remain that I am
planning to have a look at.

Tamas_Berghammer · April 20, 2017, 2:01pm

AFAIK the Ubuntu 14.04 cmake builder runs tests using ToT clang (built on the build bot) as step test3 and test4 and it seems to be green so if you are seeing different result then I would expect it to be caused by a configuration difference between the setup the bot has and you have (or the bot runs the tests incorrectly).

Pavel_Labath · April 20, 2017, 2:03pm

Yes, that is my assumption. I'd like to see the tests that are failing (and
how) to see if we can eliminate the system dependencies from these tests...

Scott_Smith · April 20, 2017, 6:30pm

Top of tree clang has the same set of unexpected successes.

Scott_Smith · April 20, 2017, 6:42pm

Sorry, I take that back. I forgot to save the buffer that ran the test script. Oops

I get a number of errors that make me think it’s missing libc++, which makes sense because I never installed it. However, I thought clang automatically falls back to using gcc’s libstdc++.

Failures include:

Build Command Output:
main.cpp:10:10: fatal error: ‘atomic’ file not found
#include
^~~~~~~~
1 error generated.

and

Build Command Output:
main.cpp:1:10: fatal error: ‘string’ file not found
#include
^~~~~~~~
1 error generated.

Pavel_Labath · April 20, 2017, 9:34pm

It does, however, we have tests which explicitly request libc++ to test the
relevant pretty-printers.

That said, the tests should detect libc++ presence and get skipped if you
don't have it, but that could have gone wrong. Can you send me the exact
list of tests which are failing (and also make sure to sync to master, as I
have fixed a couple of these today).

thanks,
pl