FileCheck feature request- by default ignore IR-"headers"

Hi all-
I hope this is the right place for this.

Anyway, the primary usage for the utility "FileCheck" is to pattern match for specific values in a stream. This is perfectly consistent and deterministic for the most part! However, when validating clang's LLVM-IR generation, it is possible to make an invalid match against the top few generated lines (the LLVM-IR Headers), which are inconsistent.

For example (from my personal pain), we have a test: clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the word "macros". Unfortunately, my SVN workspace name was "ms_macros", which gave me a false match. I'm a bit ashamed to say how long this took me to figure out :slight_smile:

Anywya, to my feature request: to prevent someone else from going through this pain again, would it be possible to have FileCheck ignore these first 4 lines as default behavior? I suspect there might be a few tests that would want to validate those headers, though I expect those are the vast minority, so I would think this should be opt in.

-Erich

"Keane, Erich via llvm-dev" <llvm-dev@lists.llvm.org> writes:

I hope this is the right place for this.

Anyway, the primary usage for the utility "FileCheck" is to pattern
match for specific values in a stream. This is perfectly consistent
and deterministic for the most part! However, when validating clang's
LLVM-IR generation, it is possible to make an invalid match against
the top few generated lines (the LLVM-IR Headers), which are
inconsistent.

For example (from my personal pain), we have a test:
clang/test/CodeGen/debug-info-macro.c that has a 'check-not' on the
word "macros". Unfortunately, my SVN workspace name was "ms_macros",
which gave me a false match. I'm a bit ashamed to say how long this
took me to figure out :slight_smile:

This is a bug in the test and an annoyingly common pitfall when it comes
to writing "CHECK-NOT" checks - it should be checking for something more
specific.

Anywya, to my feature request: to prevent someone else from going
through this pain again, would it be possible to have FileCheck ignore
these first 4 lines as default behavior? I suspect there might be a
few tests that would want to validate those headers, though I expect
those are the vast minority, so I would think this should be opt in.

Unfortunately this isn't really practical to do - FileCheck is
completely unaware of what kind of file it's checking, so it wouldn't
know when to skip those couple of lines (many tests are not IR!).

Yeah, this is a documented shortcoming of FileCheck:
http://llvm.org/docs/TestingGuide.html#fragile-tests

I think the most practical thing to do would be to make FileCheck reject test cases that consist only CHECK-NOT directives, unless a flag is passed to disable this behavior.

Alright, I guess it isn’t just my pain then, it makes it feel better J I think that proposed feature would be really nice, since it would encourage people to write tests that have a //CHECK: some-thing-after-header first!

re-add llvm-dev…

FWIW - a -NOT only test looks especially problematic to me. The test would pass if the program produced no output, basically - which sounds overly loose (a shade less problematic than the “run this and don’t check anything - not crashing is all we’re testing for here” - which should have some positive test of the behavior that is expected other than crashing).

So I’d see such a constraint as potentially interesting on that basis - if it fixes this other issue as well, eh, helpful.

But I’m also OK with it being a thing we document, and every new contributor eventually learns once… it doesn’t seem to have a lot of overhead.