Hi,
TLDR
FileCheck just became less noisy on failures by default. However, for
some cases, it now provides too little debug info, so new
configuration options have landed. When debugging CI failures, many
LLVM committers have no feasible way to adjust these options, so I
think LLVM CI configs should include the following environment setting
to maximize the debug info:
FILECHECK_OPTS='-dump-input-filter=all -vv -color'
Background: FileCheck Input Dumps by Default
D81422 recently landed to enable -dump-input=fail by default in
FileCheck. As a result, when FileCheck fails, it by default produces
an annotated input dump like the following (which is intended to be
viewed in a monospaced font):
<<<<<<
1: foo
2: foo
3: foo
4: foo
5: foo
6: foo
7: foo
8: foo
9: foo
10: foo
11: foo
12: foo
13: foo
14: foo
15: hello world
16: foo
check:2'0 X~~ error: no match found
17: foo
check:2'0 ~~~
18: foo
check:2'0 ~~~
19: foo
check:2'0 ~~~
20: foo
check:2'0 ~~~
21: foo
check:2'0 ~~~
22: foo
check:2'0 ~~~
23: foo
check:2'0 ~~~
24: foo
check:2'0 ~~~
25: foo
check:2'0 ~~~
26: foo
check:2'0 ~~~
27: foo
check:2'0 ~~~
28: foo
check:2'0 ~~~
29: foo
check:2'0 ~~~
30: goodbye word
check:2'0 ~~~~~~~~~~~~
check:2'1 ? possible intended match
31: foo
check:2'0 ~~~
32: foo
check:2'0 ~~~
33: foo
check:2'0 ~~~
34: foo
check:2'0 ~~~
35: foo
check:2'0 ~~~
36: foo
check:2'0 ~~~
37: foo
check:2'0 ~~~
38: foo
check:2'0 ~~~
39: foo
check:2'0 ~~~
40: foo
check:2'0 ~~~
> ```
>
> LLVM developers' views on this change vary. See the following
> discussions:
>
> * <[https://reviews.llvm.org/D81422](https://reviews.llvm.org/D81422)>
> * Thread starting at: <[http://lists.llvm.org/pipermail/llvm-dev/2020-June/142369.html](http://lists.llvm.org/pipermail/llvm-dev/2020-June/142369.html)>
>
> ## The Good ##
>
> Some people feel this change is a significant improvement for
> debugging FileCheck errors. It's especially helpful for those who
> aren't aware of the `-dump-input=fail` option or the `FILECHECK_OPTS`
> environment variable, which can be used to set such debugging options.
> This change is perhaps even more helpful in the case of CI
> configurations that don't set `-dump-input=fail`. In that case, upon
> a test failure, it is not feasible for many LLVM committers to set
> `-dump-input=fail` and run a failing test again on precisely the same
> platform in order to get the debugging info they need. Now they don't
> have to.
>
> ## The Bad ##
>
> Others feel this dump produces far too much noise to be useful,
> especially when the FileCheck input is large and the test suite is
> being run in a terminal window. Sometimes the actual error message is
> buried thousands of lines earlier in a terminal's scrollback or may
> have even overflowed the scrollback buffer.
>
> # Fix 1: Filtered Input Dumps #
>
> In an attempt to satisfy both views, D82203 just landed to filter the
> input dump to just error diagnostics plus some context. Thus, the
> previous example now looks like the following by default:
>
> ```
.
.
.
11: foo
12: foo
13: foo
14: foo
15: hello world
16: foo
check:2'0 X~~ error: no match found
17: foo
check:2'0 ~~~
18: foo
check:2'0 ~~~
19: foo
check:2'0 ~~~
20: foo
check:2'0 ~~~
21: foo
check:2'0 ~~~
.
.
.
25: foo
check:2'0 ~~~
26: foo
check:2'0 ~~~
27: foo
check:2'0 ~~~
28: foo
check:2'0 ~~~
29: foo
check:2'0 ~~~
30: goodbye word
check:2'0 ~~~~~~~~~~~~
check:2'1 ? possible intended match
31: foo
check:2'0 ~~~
32: foo
check:2'0 ~~~
33: foo
check:2'0 ~~~
34: foo
check:2'0 ~~~
35: foo
check:2'0 ~~~
.
.
.
>>>>>>
The Good
Unless there are many FileCheck errors (e.g., one FileCheck run can
report an error per CHECK-LABEL), the input dump is now relatively
small. Thus, this change supports the original motivation to enable
-dump-input=fail by default but avoids being overwhelming when the
full input is large.
The Bad
I find that it is often true that the default amount of context, as
shown in the example above, is far from sufficient. For example,
often there is no possible intended match or it’s a bad guess, and
the intended match is hundreds of lines away from any error
diagnostic. Even worse, the actual bug is often at a different
directive than the directive for which FileCheck reports an error. In
other words, the information I need is often filtered out, so I need
to be able to configure the input dump filter.
Fix 2: Filter Configuration
D83097 and D82203 have landed to implement -dump-input-filter and
-dump-input-context for configuring the filtered input lines (that
is, include more than just errors) and their lines of context. See
FileCheck -help for details.
Where to Specify Them
Like other FileCheck debugging options, these can be set via
FILECHECK_OPTS either globally (e.g., in a ~/.profile) or per test
suite run (e.g., FILECHECK_OPTS=-dump-input-context=100 ninja check). They can even be set as command-line options in specific
FileCheck calls within specific tests in case those calls have very
specific debugging needs. In that case, they won’t conflict with
options in FILECHECK_OPTS because multiple occurrences of an option
are resolved in favor of the option requesting the most debugging
info.
Is This Too Configurable?
It has been suggested that the configuration space provided by
-dump-input-filter and -dump-input-context is too large. We could
have instead reduced it to a simple boolean: filter as shown above by
default, or show the full input if requested. However, I anticipate
that the default amount of context, no matter what it is, will be too
frequently wrong for some LLVM developers’ use cases. On the other
hand, a full input dump can be overwhelming even when enabled for a
single test. I think the ability to select levels in between these
two extremes is important. Finally, in my opinion, this extra
configurability adds minimal complexity to the FileCheck
implementation. Most of the implementation needed is already needed
to support just the default configuration.
Fix 3: Filter Configuration in CI
As I mentioned above, upon a test failure in CI, it is not feasible
for many LLVM committers to set FILECHECK_OPTS and run the failing
test again on precisely the same platform in order to configure the
precise information they need for debugging. In my experience, the
inability to request sufficient debugging information is a much more
severe problem than encountering too much debugging information. For
these reasons, I think LLVM CI configs should maximize the debugging
info upon FileCheck failures by setting:
FILECHECK_OPTS='-dump-input-filter=all -vv -color'
I use this successfully in my local Gitlab CI configs. Hopefully
others will adopt it too.
Thanks.
Joel