RFC: Improving lit's debug output

LLVM Project
44

Agreed.

With the patch I suggested applied, here is an example execution trace when the external shell is bash:

+ echo 'RUN: at line 1:' 'echo '\''hello world'\'' | FileCheck bogus.txt && echo success'
RUN: at line 1: echo 'hello world' | FileCheck bogus.txt && echo success
+ echo 'hello world'
+ FileCheck bogus.txt
Could not open check file 'bogus.txt': No such file or directory

I’ve thought of a few possible solutions:

  1. Drop set -x entirely
    • New execution trace:
      : 'RUN: at line 1'; echo 'hello world' | FileCheck bogus.txt && echo success
Could not open check file 'bogus.txt': No such file or directory
    • Benefit: Much less clutter on stderr. There would be no + lines. There would be only the full RUN: at line N: command lines plus anything they print to stderr. We could even change it to : 'RUN: at line N'; command (as shown above) so the whole thing could be copied and pasted into a terminal. Then, multiple RUN lines could be copied at once, as long as the commands wrote nothing to stderr in between.
    • Problem: Execution tracing within each RUN line will be lost. That can make it hard to see which commands were executed and which failed in a series joined by &&, ||, |, etc. Given that we’re talking about external shells, there might be more complex shell constructs we’d like to trace too, like (...), {...}, $(cmd), etc.
    • Workaround: The user can add RUN: set -x to a test when they need the full execution trace. I suspect they would not do this permanently in a test as that would circumvent our goal here. Instead, I imagine they would add it temporarily when needing to debug a failed test, but that’s less helpful when debugging failed tests from unattended runs, in CI for example.
  2. Replace echo with :

    • New execution trace:

      + : 'RUN: at line 1:' 'echo '\''hello world'\'' | FileCheck bogus.txt && echo success'
+ echo 'hello world'
+ FileCheck bogus.txt
Could not open check file 'bogus.txt': No such file or directory

    • Benefit: We lose only the echoed RUN: at line N: command but not its corresponding + line or any other + line.

    • Problem: Quoting must be removed from 'command' in the remaining : 'RUN: at line N:' 'command' when copying it to a terminal. That’s not so hard if command is simple, but it’s going to be tedious and error-prone if command itself contains special characters (as in the above example). For this reason, I think I dislike this solution the most.

  3. Use a trick like { set +x; } 2> /dev/null and { set -x; } 2> /dev/null

    • New execution trace:

      RUN: at line 1: echo 'hello world' | FileCheck bogus.txt && echo success
+ echo 'hello world'
+ FileCheck bogus.txt
Could not open check file 'bogus.txt': No such file or directory

    • Benefit: We could use that to suppress only the + echo 'RUN: at line N:' 'command' line while keeping the echoed RUN: at line N: command and all other + lines. The {...} 2> /dev/null part avoids seeing tracing of the set command itself, which would just replace one form of clutter with another.

    • Potential problem: Is it portable to all sh shells? It seems to work for /bin/sh (dash) and /bin/bash on my system. Is there a corresponding windows cmd trick, or will we have to invent another solution there [edit: @echo is the solution there]? If you like this solution, please try it out at a few shell prompts on your system to confirm it works. Here’s a full test with expected output:

      $ /bin/sh # or whatever shell
$ set -x
$ echo before; { set +x; } 2> /dev/null; echo a RUN line; { set -x; } 2> /dev/null; echo after
+ echo before
before
a RUN line
+ echo after
after

If I were deciding by myself, I’d probably try to land 1 first because it seems to strike a nice balance between ease of implementation and usability while still pushing us to migrate to lit’s internal shell for a better execution trace. But please give your feedback as I might have missed important use cases again.

I realize there’s some subtlety in the above descriptions, and I’m happy to answer questions if I’m not being clear.

2 Likes