RFC: Improving lit's debug output

[MaskRay]

@jdenny-ornl

Lit can be configured to execute RUN lines using either lit’s internal shell or an external shell (e.g., bash, sh, window’s cmd). Lit’s internal shell is usually best for test portability because it’s implemented in python. To configure a test suite to use lit’s internal shell, make sure the following line appears in the test suite’s lit.cfg or other configuration file:

config.test_format = lit.formats.ShTest(execute_external=False)

Context to readers who are unfamiliar (I was in this category before I read some code of lit to try reviewing some patches):

• By default, Windows use lit’s internal shell and non-Windows platforms use their external shells. The internal shell uses $ as a command prompt (under Command Output) while external shells may use +
• LIT_USE_INTERNAL_SHELL=1 path/to/llvm-lit ... uses the internal shell. Another way is lit.formats.ShTest(execute_external=False)
• REQUIRES: shell disables tests that use the internal shell. https://github.com/llvm/llvm-project/blob/0f8bab8d590ed0cb5402f72009b32cbad115f013/llvm/utils/lit/lit/llvm/config.py#L57

% rg 'REQUIRES:.*shell' -l | wc -l
165

I am in favor of the the formatting change (⚙ D156954 [lit] Improve test output from lit's internal shell), even if it is specific to the internal shell.

I am also in favor of ⚙ D154984 [lit] Drop "Script:", make -v and -a imply -vv , as path/to/llvm-lit -v a.test now tells the failed command. Previously I always use -vv to tell the failed command (Script: output lists all commands; when there are many, it’s unclear which one failed).