One of the issues we’ve faced a couple of times (and will continue to face) is that lit is generally built around substitutions and command lines, but different compilers will generally have different substitutions, and worse – different command line syntaxes.
The most recent example of this is the stop-hooks test that was added which uses the %cc and %cxx substitutions. This will work fine with GCC or clang, but not with clang-cl, where we need an entirely different command line.
If we’re going to grow this test suite, I think we’re going to need a solution to this.
The main issue I want to address here is that we need a way to abstract over the difference between command line syntaxes. gcc and clang are pretty similar, but they can occasionally differ in minor ways. But clang-cl will always be different.
One idea I’ve had here is to extend lit with the ability to create our own custom prefix commands (similar to RUN lines), but where we can provide lit with the prefix and a function to call to execute it. The thinking being that we can add something like a COMPILE and LINK command, which you could invoke like this:
// COMPILE: source=%p/Inputs/foo.cpp
// COMPILE: opt=none
// COMPILE: compiler=default
// COMPILE: out=%t.obj
// COMPILE: link=no
// LINK: obj=%t.obj
// LINK: linker=default
// LINK: nodefaultlib
// LINK: entry=main
// LINK: out=%t.exe
Here “default” means whatever lit decides, which would usually depend on how you configured your CMake. But for some tests you could specify an explicit compiler, for example, you could say compiler=gcc or compiler=msvc, and the test would fail if those compilers are not configured.
This is actually very similar in spirit to how dotest.py’s “builders” work, but extended to lit.
If we go this route, the first step would be to extend lit with a general notion of pluggable commands, independently of LLDB.
After that, we could implement the COMPILE and LINK commands in LLDB, perhaps even reusing much of builder.py from the dotest sutite.
Note that I’m not attempting to address the idea of running the test suite with different compilers in the same run. I have further ideas for how to address that, but I don’t think we need to do that right now as it’s not an immediate need.
Does anyone have any thoughts on this?