Bad test health

Project Infrastructure LLVM Dev List Archives
1

I was curious how widespread bad usage of FileCheck prefixes is in the
LLVM/Clang test suite. Running the attached script yields over 200 tests
(~1.5%) which are essentially broken because they contain checks that
are never checked.

Most widespread is the usage of the "CHECK" prefix while all FileCheck
invocations have an explicit check prefix. Then there are quite a few
prefixes that have no corresponding FileCheck invocation. And of course
lots of typos. Oh, and some tests don't even call FileCheck in the first
place!

There are also a lot of different ways people disable checks: some use
explicit CHECK-DISABLED or CHECK-FIXME, others use HECK, CHECKX or
similar, often without a comment. Since there are also typos with the
same spellings and some tests are already broken with corrected typos
this makes it quite cumbersome to fix.

Sadly I can't come up with non-invasive changes to prevent these
mistakes. And since FileCheck's syntax is so common in comments the rate
of false positives is too high for automatic checking.

So I at least want to bring this issue to the attention of others.

-Nico

test_lint.py (4.14 KB)

2

That’s probably because it’s a regression test for a crasher. It’s checking that the compiler exited successfully.

—Owen

3

I assume Nico means it's a test with CHECK lines but doesn't invoke
FileCheck.

Though personally I object to tests that don't check anything other than
"doesn't crash" - if a certain example previously crashed and now doesn't,
there was some codepath that was untested (its output unconstrained). The
crash just revealed the untested codepath, testing that it doesn't crash
still leaves the behavior mostly unconstrained to "do anything other than
crashing" which seems insufficient. We should take the opportunity to test
the untested behavior that's been discovered.

- David