[moving this to lldb-dev for more visibility.]
Sorry, I was in a hurry yesterday, so I did not explain myself fully. Let me try to elaborate.
What I am trying to avoid here is getting useless noise in build automation where test cases proclaim their failure, which however tells us nothing of value as the issue is simply “libc++ not available” vs. “data formatters broken” That is an ability I would strongly like to preserve
I agree that we should have an ability to skip libc++ tests if the library is not available (similarly for go, etc.). However, I believe there should be as little “magic” in that as possible. Otherwise you run into the opposite problem where a test passing could mean “everything is ok” or “our auto-skipping magic has gone wrong”, which I would argue is a worse situation than the first one.
Currently, we have a lot of magic in here:
- self.build() silently builds an executable with a different library even though libc++ was requested. (otherwise you wouldn’t even be able to list the modules of the executable)
- the test decides to skip itself without giving any indication about (sure, it shows up in the “skipped tests” list, but a lot of other stuff appears there as well. and I would argue that this is a different situation: usually we do skips based on the OS, architecture, or other “immutable” characteristics. Presence of libc++ is something that can be usually changed by simply installing a package.)
I’d like to propose a few alternative solutions to achieve both of these objectives:
a) Use the existing category system for this: libc++ tests get tagged as such and the user has to explicitly disable this category to avoid running the tests. Potentially, add some code to detect when the user is running the test suite without libc++ installed and abort the run with some message like “No libc++ detected on your system. Please install libc++ or disable libc++ tests with
I like this solution because it is easily achievable and has no magic involved. However, it requires an action from the user (which I think is a good thing, but I see how others may disagree). That’s what I meant for asking about your “use case”. Would you be able to fit it inside this framework?
b) Use category tagging as in (a), but auto-disable this category when libc++ is missing and print a big fat warning “No libc++ detected, libc++ tests will not run”. What is different from the current state is that this libc++ detection would work centrally, which would give us the possibility to print the warning (e.g. right before the “Ran XXX test suites” message")
I like this a bit less, as it is still automatic, but I am fine with it, as it would give a clear visual indication of what is happening.
What do you think? Do you have another proposal?
I can implement either of those, but I was to get some feedback first…