[RFC] Changing `lit` default output

Project Infrastructure LLVM Dev List Archives
1

Hi all,

I have some patches up in:
http://reviews.llvm.org/D20721
to make lit output more readable when commands fail. I’d like to get a little more feedback on this before landing, since it impacts how test failures appear when using the internal shell script runner (which is not the default so this will currently mostly impact Windows or test frameworks that intentionally set this behavior).

These patches change the output of a script like true && echo hi && false to something like:

$ true
$ echo hi
hi

$ false
note: command had no output on stdout or stderr
error: command failed with exit status 1

instead of the old:

Command 0: "true"
Command 0 Result: 0
Command 0 Output:

Command 0 Stderr:

Command 1: "echo" "hi"
Command 1 Result: 0
Command 1 Output:
hi

Command 1 Stderr:

Command 2: "wc" "missing-file"
Command 2 Result: 1
Command 2 Output:
None

Command 2 Stderr:
None

which I think is significantly more readable.

One change in the patches is that it will also start including, inline, the output of files which were redirected. This solves a common problem where a command fails when it wasn’t expected to, and its output was captured. In such cases the test log just showed the command failure, but since the output was redirected you couldn’t tell from the log why the common failed.

Please let me know if you have any objections to the new format.

  • Daniel
2

What would the output of "echo '$ echo hi ’ " be and how do you differentiate from ‘$ echo hi’ ?

3

(+ the real cfe-dev)

4

As long as the individual commands are still copy-pastable into both cmd and bash, I’m happy. I think practice this means the output should look more like this:

some test:
// RUN: %clang_cc1 %s -verify

output:
$ “C:\src\llvm\build\bin\clang.exe” -cc1 “C:\…\test.cpp” -verify

Essentially every program used in the LLVM test suite understands forward slashes, so we could rewrite the path to test.cpp to use forward slashes, but I think cmd wants to see backslashes in argv0. Then our output would look like:
$ “C:\src\llvm\build\bin\clang.exe” -cc1 C:/src/llvm/…/test.cpp -verify

5

This part I didn’t change, but I’m not completely sure I understood your message – is there a change you wanted here?

  • Daniel
6

If you're not changing the quoting, then everything is fine, we don't have
to do anything.

I was trying to suggest that we canonicalize path arguments to use forward
slashes to reduce the need for quoting, but it should really be done as a
separate change.

7

Hi all,

I have some patches up in:
  http://reviews.llvm.org/D20721
to make `lit` output more readable when commands fail. I'd like to get a
little more feedback on this before landing, since it impacts how test
failures appear when using the internal shell script runner (which is not
the default so this will currently mostly impact Windows or test frameworks
that intentionally set this behavior).

These patches change the output of a script like `true && echo hi &&
false` to something like:

What would the output of "echo '$ echo hi ' " be and how do you
differentiate from '$ echo hi' ?

"Don't do that"?

Actually, the output I gave wasn't complete, what we actually currently
show is:

$ "echo" "hi"
# command output:
hi

but of course your comment still applies. However, the goal is to make for
a readable output for the majority of use cases, not necessarily to be
unambiguous.

- Daniel