Changing behavior of lit.py's -v flag

Hi,

I think lit.py’s current behavior is somewhat unintuitive in the presence of -v (and not
using -vv), so I’m proposing that it be changed.

Current behavior:
-v: Prints all the substituted lines, not clear which line failed :frowning:
-vv: Prints all the lines up to and including the line that failed :slight_smile:

Option 1:
-v: Prints only the failing line :slight_smile:
-vv: Same as today. :slight_smile:
-vvv (new): Prints all the substituted lines, highlighting* the line that failed :slight_smile:

Option 2 (less deviation from status quo):
-v: Adopts behavior of -vvv from Option 1. :slight_smile:

-vv: Same as today. :slight_smile:

See also: https://twitter.com/brentdax/status/1154955239620956160 (note: my opinion has
changed slightly since my comment there)

Varun

  • highlighting — could be bold and/or underlined or something else. Based on some simple
    testing on a Mac terminal, this doesn’t break copy-pasting even if you’re using the default
    formatting-aware copy-paste.

Hi Varun,

I'm definitely in favour of making -v more useful like this.

Option 2 (less deviation from status quo):
-v: Adopts behavior of -vvv from Option 1. :slight_smile:
-vv: Same as today. :slight_smile:

I think I prefer this one. There's a significant minority of tests
where a the single failing line isn't enough to work out what's
actually happened, if the FileCheck is a separate RUN for example.
Under this scheme no-one even needs to know -vv exists (which is good,
because I've been working with LLVM for many years and didn't until
this thread).

Cheers.

Tim.

Tim Northover via llvm-dev <llvm-dev@lists.llvm.org> writes:

Hi Varun,

I'm definitely in favour of making -v more useful like this.

Option 2 (less deviation from status quo):
-v: Adopts behavior of -vvv from Option 1. :slight_smile:
-vv: Same as today. :slight_smile:

I think I prefer this one. There's a significant minority of tests
where a the single failing line isn't enough to work out what's
actually happened, if the FileCheck is a separate RUN for example.
Under this scheme no-one even needs to know -vv exists (which is good,
because I've been working with LLVM for many years and didn't until
this thread).

+1.

                      -David

Tim Northover via llvm-dev <llvm-dev@lists.llvm.org> writes:

Hi Varun,

I’m definitely in favour of making -v more useful like this.

Option 2 (less deviation from status quo):
-v: Adopts behavior of -vvv from Option 1. :slight_smile:
-vv: Same as today. :slight_smile:

I think I prefer this one. There’s a significant minority of tests
where a the single failing line isn’t enough to work out what’s
actually happened, if the FileCheck is a separate RUN for example.
Under this scheme no-one even needs to know -vv exists (which is good,
because I’ve been working with LLVM for many years and didn’t until
this thread).

+1.

+1 as well.

Perhaps just alias -v to -vv. The -v option is pretty much worthless.

Making -v behave like -vv would be huge win for anyone trying to debug a failure on a bot you don’t own, and wouldn’t require bot owners to change anything.

I’ll be happy to review the patch, or write it myself if that helps.

thanks…
don

In the mean time, how about changing "ninja check" to run lit with -vv
instead of -v by default?
https://reviews.llvm.org/D72389

This gives me output like:

******************** TEST 'LLVM :: CodeGen/AMDGPU/fneg-combines.ll'
FAILED ********************
Script: