RFC: Making unit tests run by default on ninja check-lldb

Subprojects LLDB
1

Right now there are two ninja check targets: “ninja check-lldb”, which runs dotest and all of the SB API tests, and “ninja check-lldb-unit” which runs the gtest unit test suite.

I would like to make unit tests run by default. This entails two things, which could be done independently of each other.

  1. Rename check-lldb to check-lldb-python, and create a new check-lldb that depends on check-lldb-unit and check-lldb-python. This way, when you run “ninja check-lldb” you get both.

  2. Update the build bots to run both. We would probably want them as a separate step, so the existing step that runs “ninja check-lldb” would need to change to run “ninja check-lldb-python” instead. To add a unit test step, we would need to add another step that runs “ninja check-lldb-unit”.

The big unknown here is how to make the buildbots understand unit test failures and trigger a failure when ninja check-lldb-unit fails. A potential first step could be to just update the buildbot to run ninja check-lldb-python instead, and don’t add the second step, but if there’s anyone who knows how to make it parse the output of ninja check-lldb-unit and report as a failure, that would be great.

2

Hi Zachary,

3

Hi Ying,

Our dotest.py lldb test results go through that lit test parser system? I see XPASS happen frequently (and in fact is my whole reason for starting a thread on getting rid of flakey tests, or making them run enough times so that their output can be a useful signal rather than useless). According to this script, an XPASS would be listed as failure. I’m not seeing us treat XPASS as failures AFAICT.

Are we just saying that our gtests get processed by that?

-Todd

4

Yes, the output of dotest.py goes through LitTestCommand parse.
The parser is matching for “XPASS”, but dotest output is using “UNEXPECTED SUCCESS”. :slight_smile:

Thanks,
Ying

5

Oh haha okay. :slight_smile:

Thanks for explaining, Ying!

-Todd