skip some tests with "check-lldb"

Hi,

I'd like to skip some tests when I run "ninja check-lldb", because they fail.
I am on release_70 branch.
I know I could use dotest.py directly, but that would exercise only one thread.
Is there a way to execute the tests parallel on all cores and in the
same time skip some of the tests?

Thanks,
Gabor

I just realized that `dotest.py` has a --thread option. Is that the
one which is used during the lit test (`ninja check-lldb`) ?

Unless you pass --no-multiprocess to dotest, it should detect how many cores your system has and use them.

That’s okay, but is it possible to skip a few tests, when using lit? I was thinking about moving the test files I want to skip, but that has obvious drawbacks. Also --filter does not seem so useful in this case.

When using lit as the driver (which is the case for check-lldb), the test
granularity is at file-level: lit invokes dotest.py for every test file. You
should be able to specify files to skip with lit’s --filter.

With regards to threading, lit will schedule one instance of dotest.py on every
thread, which processes one test file. Unless you passed specific options for
dotest.py, the latter will run the different tests within that one file on
different threads as well. (IIRC)

Hi Jonas,

Thanks for the clarification.
Finally I could skip some of the tests by using a negative lookahead
regex passed to --filter. I wouldn't say that this is so convenient
and easy to overview, but works.
For example, to skip the TestCalculatorMode.py and TestPrintArray.py
tests I used the following argument (appended to the normal lit
command line reported by "ninja check-lldb -v" ) :
--filter '^((?!(TestCalculatorMode\.py|TestPrintArray\.py)).)*$'
Would be beneficial though to have a --exclude besides --filter to
avoid using a complex negative lookahead regex.

Cheers,
Gabor

Hi Jonas,

Thanks for the clarification.
Finally I could skip some of the tests by using a negative lookahead
regex passed to --filter. I wouldn't say that this is so convenient
and easy to overview, but works.
For example, to skip the TestCalculatorMode.py and TestPrintArray.py
tests I used the following argument (appended to the normal lit
command line reported by "ninja check-lldb -v" ) :
--filter '^((?!(TestCalculatorMode\.py|TestPrintArray\.py)).)*$'
Would be beneficial though to have a --exclude besides --filter to
avoid using a complex negative lookahead regex.

I have to agree that this looks overly complex. Maybe it's something to bring
up on the llvm-dev mailing list as a general improvement for lit?

When I am working on some changes and there are tests I know I've broken but don't want to fix just yet, I just edit the test file and skip it using the skip decorators. You have to remember not to check in the skips when you check in whatever change you are making - but presumably by the time you are ready to submit something you will have fixed the tests anyway. That's much simpler than trying to monkey with the test runner to do your bidding.

Jim