Currently, LLD tests only use llvm-mc / llvm-as to generate test inputs. This is generally the right level of abstraction for the things we want to test. However, sometimes feeding C++/ObjC inputs into clang is simpler. For example, the Mach-O linker backend needs to manipulate ObjC structs, so piping ObjC test inputs into clang+lld makes for a compact, easily modifiable test.
Thus far, we have worked around the issue by taking clang’s output, running it through obj2yaml, and committing the yaml to the repo. We include a block comment in the test itself containing the original C++/ObjC code, so that future readers can modify / update the test. However, this is fairly tedious, and opens up the possibility of the code in the comments getting out of sync with the committed YAML.
Moreover, future versions of clang may produce different assembly outputs given the same inputs, making the tests even harder to update, especially if discovered at a much later date. Having clang be part of the test suite input generators means that if clang’s changes breaks an LLD test, we discover it immediately.
So those are the pros. The main con I can think of is that this brings in a very big dependency to our test suite, slowing the dev cycle. I’m thinking we could mitigate this by splitting out clang+LLD tests into its own “check-lld-with-clang” target. Would love to hear y’alls thoughts.
There’s cross-project-tests that is designed for supporting tests that depend on multiple components of the LLVM project (e.g. clang + LLD, although I don’t know immediately if there are any concrete examples that use that combination in those tests yet), so there should be no need for another check target.
Whilst there is certainly a benefit to integration tests which test that the pipeline source -> final output produces some desired behaviour, I’d be careful about when you use clang to generate LLD test inputs, as that will cause the LLD inputs to change somewhat frequently. It might be that a clang change causes you to lose some test coverage of something in your intermediate object file that you care about. Note that your linker will presumably still need to support the older object file style, even if clang no longer generates it, since you could come across such an object later on, so you still need test coverage for it.
I agree with jh7370. Having worked on a (non-LLVM) linker that did permit compiler based tests I appreciate the predictability of object file that comes from assembler and yaml2obj. I also find that the pain of writing lots of assembler helps keep the tests small and too the point. While I think that these could be segregated within a sub-directory of the lld/test directory I think the cross-project-tests are a better place.
The other difficulty of adding clang as a dependency is imposing that on non-clang based projects that use LLD as their linker.
my $0.02: I’d also prefer having clang in the set of test tool rather than having to do the input generation manually.
But, as explained in the previous two comments, it would be great to not have people abuse it too much. (Maybe soemthing like “only use clang in cases where simple asm is not sufficient for the test case”?)
If you need to, just use Clang to generate the assembly you commit (and perhaps include a comment with the source somewhere). But really you shouldn’t need to write in C, pretty much every linker test should be writable in a short amount of assembly. We used to have tests in our fork written in C (perhaps we still do) and they cause pain when merging newer versions. Having linker tests changing because Clang changed is just not a good place to be; it’s much harder to even see if the test is still testing what was originally intended.