Clang codegen tests and LLVM passes


This was brought up several times now, but looks like it needs to be
mentioned more globally. Clang has many (~1516) front-end codegen tests.
Out of those tests, 364 tests are likely "broken":
$ clang/test$ grep -rl " -O[1-3]" | wc -l

Clang codegen tests should not be checking anything more than
clang IR codegen itself. They should not be testing the asm output.
They should not be invoking any middle-end optimization passes.

I suspect some of those 364 tests are as fragile.
It would be great if no new tests would have this design issue.
It would also be great to reduce the number of such tests.

In particular, this time i stumbled into this with
that broke clang/test/CodeGen/arm_acle.c
Given the nature of that test (it doesn't just greedily check too much, so it
could be relaxed to still pass, it basically contains end-to-end testing
for certain ARM intrinsics) so it can't really be relaxed.
I had to disable that test as i'm not sure how it should be fixed.


It’s an issue, but it’s not as bad as that. Many tests use -O1 -disable-llvm-passes to get “optimized” IR output from clang which is slightly different. This matters for lifetime markers, for example.

I tried this query and got these results:
$ git grep -l " -O[1-3]" …/clang/test | xargs grep -L -l ’ -disable’ | wc -l

So, 99/~1516 → ~6.5% tests use the bad pattern. Some of the tests are truly integration tests, checking that clang emits an IR pattern that LLVM can optimize, but I think most are not.

I think the real issue here is that writing CHECK lines for clang’s IR output is a pain, so people lean on the optimizer to clean up the IR first so they can match something that’s more logical. If we did something to improve that, the situation wouldn’t be as bad.