Super-verbose failure mode for FileCheck

Hi All,

FileCheck is a great tool for writing tests.
However, it can be also a great tool for hiding the error cause from you, especially if a very common idiom

RUN: ./myprog | FileCheck %s

is used.
If you need to know the output of “myprog” to figure out why your test is failing — then well, good luck,
you would need to manually copy’n paste that line, run it, and then note the output.
(produced line “expected/got” are rarely enough to debug the problem)

Would it be useful to have an optional, off-by-default mode for FileCheck, turned on by an environment variable,
which would save all the input it gets, and then dump it on failure?
I also wish we could get a

The test case I imagine would be to run:

ninja check-X

note the failure,
and then rerun:

env FILECHECK_SUPER_VERBOSE=1 ninja check-X

Thoughts?
George

George Karpenkov via llvm-dev <llvm-dev@lists.llvm.org> writes:

Would it be useful to have an optional, off-by-default mode for
FileCheck, turned on by an environment variable, which would save all
the input it gets, and then dump it on failure?

Yes please! I'd also like an option to dump it to stdout. I run tests
inside an IDE and it would be convenient to be able to look at the
output there rather than having to open new files.

                         -David

Could be useful - as Paul mentioned, having it on standard output could be good - especially for buildbots (where rerunning the command is impractical). Though a more generalized solution for ‘lit’ where it dumps all the inputs and outputs (well, I guess, the textually representable ones - maybe that’s basically just FileCheck) in a way that buildbot could split them out into separate files (easier to read, download/use, etc than one big stream), would be extra super awesome.

https://reviews.llvm.org/D49328