[RFC] Cross-project lit test suite

Reid and Amy might have some context for Windows (though I don't know
if any Windows folks have done much with this test suite).

The debuginfo-test tests are failing for a long time for us too. I won't have much time to fix them in the short term, but here's the errors I'm seeing (James do you see the same thing on your end?):

Failed Tests (7):
  debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_program_state.cpp
  debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_step_kinds.cpp
  debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_step_order.cpp
  debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_watch_type.cpp
  debuginfo-tests :: dexter/feature_tests/commands/penalty/expect_watch_value.cpp
  debuginfo-tests :: dexter/feature_tests/commands/penalty/unreachable.cpp
  debuginfo-tests :: dexter/feature_tests/unittests/run.test

The first ones seem to be related to a missing Python library in my installation:

F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe" "F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py" list-debuggers

dbgeng [dbgeng] YES (1)
lldb [lldb] NO (The system cannot find the file specified ["lldb.exe"])
vs2015 [Visual Studio 2015] NO (No module named 'win32com')
vs2017 [Visual Studio 2017] NO (No module named 'win32com')
vs2019 [Visual Studio 2019] NO (No module named 'win32com')

Which in turns generates this error:

F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe" F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py test --fail-lt 1.0 -w --builder clang-cl_vs2015 --debugger dbgeng --cflags "/Zi /Od" --ldflags "/Zi" -- F:\aganea\llvm-project\debuginfo-tests\dexter\feature_tests\commands\penalty\expect_watch_type.cpp
expect_watch_type.cpp: nan/nan (nan) [F:\aganea\llvm-project\debuginfo-tests\dexter\dex\builder\scripts\windows\clang-cl_vs2015.bat: failed with returncode 1. : The system cannot find the path specified.]

The last failure (dexter/feature_tests/unittests/run.test) is related to slash vs. backslash:

F:\aganea\llvm-project>"C:/Program Files/Python39/python.exe" "F:/aganea/llvm-project/debuginfo-tests\dexter\dexter.py" "--unittest=show-all"

test_get_script_environment (builder.Builder.TestBuilder) ... ok
test_parse_bad_whitespace (command.ParseCommand.TestParseCommand)
Throw exception when parsing badly formed whitespace. ... ok
test_parse_empty (command.ParseCommand.TestParseCommand)
Empty files are silently ignored. ... ok
test_parse_escaped (command.ParseCommand.TestParseCommand)
Escaped commands are ignored. ... ok
test_parse_good_whitespace (command.ParseCommand.TestParseCommand)
Try to emulate python whitespace rules ... ok
test_parse_inline (command.ParseCommand.TestParseCommand)
Commands can be embedded in other text. ... ok
test_parse_multi_line_comment (command.ParseCommand.TestParseCommand)
Multi-line commands can embed comments. ... ok
test_parse_share_line (command.ParseCommand.TestParseCommand)
More than one command can appear on one line. ... ok
test_add_breakpoint_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase) ... ok
test_add_breakpoint_with_source_root_dir (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
test_add_breakpoint_with_source_root_dir_slash_suffix (debugger.DebuggerBase.TestDebuggerBase) ... ok
test_get_step_info (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
test_get_step_info_no_frames (debugger.DebuggerBase.TestDebuggerBase) ... ok
test_get_step_info_no_source_root_dir (debugger.DebuggerBase.TestDebuggerBase) ... FAIL
test_did_you_mean (utils.ExtArgParse.TestExtArgumentParser) ... ok
test_PreserveAutoColors (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_auto (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_blue (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_default (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_green (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_red (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_tags (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_yellow (utils.PrettyOutputBase.TestPrettyOutput) ... ok
test_sanity (utils.UnitTests.TestUnitTests) ... ok

I get 7 failures too, but it’s not the same 7. I’m going to have a chat with some of my colleagues who work on the tool, to figure these out and will report back.

Failed Tests (7):
debuginfo-tests :: dexter/feature_tests/subtools/test/err_paren.cpp
debuginfo-tests :: dexter/feature_tests/subtools/test/err_paren_mline.cpp
debuginfo-tests :: dexter/feature_tests/subtools/test/err_syntax.cpp
debuginfo-tests :: dexter/feature_tests/subtools/test/err_syntax_mline.cpp
debuginfo-tests :: dexter/feature_tests/subtools/test/err_type.cpp
debuginfo-tests :: dexter/feature_tests/subtools/test/err_type_mline.cpp
debuginfo-tests :: dexter/feature_tests/unittests/run.test

After some digging, it turns out the 6 failures I saw (other than the unittests one), were caused by my LLVM_DEFAULT_TARGET_TRIPLE value being different to my host’s machine (I was building with a linux value, on my Windows machine, so the executables dexter was trying to build in the failing cases were unable to run). I worked around this by adding the host triple explicitly to the compiler execution.

I’ve reported this issue internally to our debug experience team, who do the dexter maintenance.

I haven’t done any Windows work with debuginfo-tests on Windows since the dexter transition. I had to disable the tests on my bot because they didn’t pass out of the box, and I never found time to follow up. It seems like other people have more leads as to what exactly is going wrong. I’d love to re-enable them, but I probably don’t have time to debug them.

I’ve gone ahead with a patch series that starts by moving the existing debuginfo-tests into a new top-level directory, with the intent that future tests can be written in other directories within the top-level one, without needing to be “debuginfo-tests”. Please take a look at the patch series and see what you think!

James

This RFC thread came up in discussion with David on https://reviews.llvm.org/D101684, where I was going to add end-to-end test testing that the WebAssembly SIMD intrinsics each lower to the correct Wasm instruction. That’s a valuable test to have somewhere because the SIMD intrinsics have a contract between the source code and the specific instructions that are emitted; in the context of the intrinsics, the IR is an irrelevant implementation detail. An end-to-end test is easily able to show that the contract is upheld, but David correctly pointed out that it doesn’t really belong in the clang directory because it tests LLVM as well.

I would propose that the cross-project-tests directory also be used to host tests that test for such contracts between the source and the emitted instructions. The two areas that come to mind are intrinsics and ABIs. Does that sound like a reasonable use case?

It does sound like a good use-case to me. There are a few places we’ve come across in the past where this sort of contract simply never gets tested directly. Often it’s the case A implies B is tested and B implies C is tested, but we actually care that A implies C. Obviously, if A instead started implying D, and D didn’t imply C, the contract would be broken, but testing wouldn’t pick it up.

Sorry, my patch series stalled due to a combination of factors. I’ll try to carve out some time in the next week or two to push it forward a bit more. Any assistance with the item discussed in https://reviews.llvm.org/D95339#2640711 (i.e. using installed tools versus those in the build directory) would of course be welcome.