[lit] XFAIL individual RUN lines?

We have a number of tests where we want to run the same code through
different subtargets:

# RUN: llc -mcpu=<subtarget1> ...
# RUN: llc -mcpu=<subtarget2> ...

Not all subtargets are expected to pass, but some will pass. We'd like
to know which tests are passing for which subtargets. We don't want to
XFAIL the whole test (because it would stop testing of working
subtargets) but we do what to know which subtargets need work.

Is it possible to somehow XFAIL an individual RUN line? The
documentation seems to imply it's all-or-nothing.

Alternatively, is it possible to get a test summary that shows PASS/FAIL
for individual RUN lines rather than for the test as a whole?

If neither of these is possible, would patches to implement them be

Thanks for your help!


Sounds a lot like

With the addition of being able to tell lit to continue if a line does
fail. I'd find both features quite useful.

I’ve wanted something similar, but for REQUIRES: lines, so that each RUN line could have a different set of required features in order to execute.


It’s not possible to XFAIL individual RUN lines any more than it is possible to mark individual lines with the UNSUPPORTED keyword (though in the latter case you could always just comment it out somehow, e.g. using the COM: directive). To do that with the current test infrastructure, you’d need to split the test into multiple files, I believe.

You could use the not command to invert the expected result, but that would cause the test to PASS rather than XFAIL, which as I understand it, isn’t what you want.

One option to extend lit might be to add some kind of --keep-going command which causes a test to fully execute even if a line in it failed. That would allow you to use the global XFAIL, but still get the testing of the subtargets (although you’d still have to display and trawl the failing test output to see what exactly failed).