Last October Mircea sent an RFC about (dis)allowing unused check prefixes in FileCheck: https://lists.llvm.org/pipermail/llvm-dev/2020-October/146162.html
(In short, if FileCheck --check-prefix or --check-prefixes specifies a prefix which does not occur in the test, FileCheck will error. Note: a
-NOT pattern is also counted as an occurrence.)
Mircea created a worksheet https://docs.google.com/spreadsheets/d/1o6q3XH1n3DDyyccnYZ_kVfhFbTDzC_S09e973_cwYuw/edit#gid=0 which may be a bit stale now but llvm-project has reached a state where all issues have been identified and fixed, or worked around (by opting in FileCheck --allow-unused-prefixes in some test directories, such as: clang/test/OpenMP, llvm/test/Transforms/Attributor and llvm/test/FileCheck).
We can make a switch if the community thinks that not allowing unused prefixes is the better default: https://reviews.llvm.org/D95849
For downstream projects using FileCheck and lit, it should be easy to restore the previous permissive behavior
from lit.llvm.subst import ToolSubst fc = ToolSubst('FileCheck', unresolved='fatal') config.substitutions.insert(0, (fc.regex, 'FileCheck --allow-unused-prefixes'))
# Note: if multiple --allow-unused-prefixes options are specified, the last wins.
Personally I am strongly in favor of disallowing unused check prefixes by default. We have identified numerous issues:
(1) typo. A misspelled check prefix does not test what it intends to test.
(2) bitrot tests due to refactoring
-NOT patterns. Sometimes a test uses something like
--check-prefixes=COMMON,FALSE to test both behaviors but they forget to include a
FALSE-NO: pattern to test that some string does not appear.
(1) and (2) are especially common. There are indeed tests where --allow-unused-prefixes is more suitable - but they are sufficiently few that I think the default should be --allow-unused-prefixes=false.
So, what do folks think?