Anyway, the primary usage for the utility "FileCheck" is to pattern match for specific values in a stream. This is perfectly consistent and deterministic for the most part! However, when validating clang's LLVM-IR generation, it is possible to make an invalid match against the top few generated lines (the LLVM-IR Headers), which are inconsistent.
For example (from my personal pain), we have a test: clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave me a false match. I'm a bit ashamed to say how long this took me to figure out
Anywya, to my feature request: to prevent someone else from going through this pain again, would it be possible to have FileCheck ignore these first 4 lines as default behavior? I suspect there might be a few tests that would want to validate those headers, though I expect those are the vast minority, so I would think this should be opt in.
"Keane, Erich via llvm-dev" <llvm-dev@lists.llvm.org> writes:
I hope this is the right place for this.
Anyway, the primary usage for the utility "FileCheck" is to pattern
match for specific values in a stream. This is perfectly consistent
and deterministic for the most part! However, when validating clang's
LLVM-IR generation, it is possible to make an invalid match against
the top few generated lines (the LLVM-IR Headers), which are
inconsistent.
For example (from my personal pain), we have a test:
clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the
word "macros". Unfortunately, my SVN workspace name was "ms_macros",
which gave me a false match. I'm a bit ashamed to say how long this
took me to figure out
This is a bug in the test and an annoyingly common pitfall when it comes
to writing "CHECK-NOT" checks - it should be checking for something more
specific.
Anywya, to my feature request: to prevent someone else from going
through this pain again, would it be possible to have FileCheck ignore
these first 4 lines as default behavior? I suspect there might be a
few tests that would want to validate those headers, though I expect
those are the vast minority, so I would think this should be opt in.
Unfortunately this isn't really practical to do - FileCheck is
completely unaware of what kind of file it's checking, so it wouldn't
know when to skip those couple of lines (many tests are not IR!).
I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior.
Alright, I guess it isn’t just my pain then, it makes it feel better J I think that proposed feature would be really nice, since it would encourage people to write tests that have a //CHECK: some-thing-after-header first!
FWIW - a -NOT only test looks especially problematic to me. The test would pass if the program produced no output, basically - which sounds overly loose (a shade less problematic than the “run this and don’t check anything - not crashing is all we’re testing for here” - which should have some positive test of the behavior that is expected other than crashing).
So I’d see such a constraint as potentially interesting on that basis - if it fixes this other issue as well, eh, helpful.
But I’m also OK with it being a thing we document, and every new contributor eventually learns once… it doesn’t seem to have a lot of overhead.