Hey all! I wound up on a sidequest while hunting something down, and found myself wishing for a lit-based harness for the tests. Michael didn’t say “that’s a terrible idea!”, so here it is. =)
Currently, check-libc relies on custom CMake commands to execute tests. Moving to lit offers several benefits:
Consistency: Matches the workflow used by Clang, LLD, and other LLVM subprojects.
Usability: Developers can use standard lit features like filters, parallelism control (-j), and verbose output (-s, -v) standardizing the debugging experience.
Reporting: Better progress reporting and failure summarization. No more digging through the cmake build output to figure out which tests failed several pages back!
Implementation Details
I have added a new custom Lit test format (LibcTest) that bridges our existing build infrastructure with lit. The custom Lit test format was needed because we don’t use GoogleTest, and many of our stdio tests require that their testdata live at a relative path. My goal here was to not change any existing tests while changing the harness.
Test Discovery: The LibcTestFormat scans the build directory for test executables. It looks for files matching the internal naming patterns libc.test.*.__unit__.__build__ and libc.test.*.__hermetic__.__build__.
Execution: It executes these binaries directly. This means we don’t need to change how tests are defined or built in CMake; we just change how they are enumerated and run.
Harness: I’ve introduced libc/utils/LibcTestFormat/ to host the python test format and added the necessary lit.cfg.py and lit.site.cfg.py.in configurations.
Trying it out
Since this is currently implemented alongside the existing system, you can try it out today without disrupting your current workflow:
Build the target:
ninja check-libc-lit
Run with llvm-lit directly:
cd <build-dir>
# Run all tests
bin/llvm-lit libc/test/
# Run specific tests
bin/llvm-lit -v libc/test/src/string/
Next Steps
If the community is happy with this direction, I’ll modify this to take over the check-libc target instead of adding a new one and then take this through full code review.
h/t Aiden Grossman for an initial review of my lit config, resulting in a much simpler approach! =)
This is something I wanted from the beginning, but I remember some complaints from the fact that downstream users may want libc to have zero dependencies on LLVM. As long as that’s not an issue I’m in support of this. Have you tested this with the platforms that use custom tools to run the tests like the GPU / RISC-V?
Thanks for the reply! I’d love to hear from others about the dependency concern. The lit stuff is already available in the runtimes build and is used in libc++, so hopefully that marks it as light-weight enough to be a reasonable choice.
I haven’t tried other platforms - just local compilation. Amusingly, I’m not sure if any system I have exposes a GPU. =) For the RISC-V case, if there’s docs, I’ll happily follow them. Otherwise, is it mostly using TEST_SUITE_RUN_UNDER to run qemu? I can be sure to test that.
I overall am supportive of this. I don’t have a large stake in libc specifically, but this will help make premerge results better for libc failures. We have nice integration with lit and worse integration for tests that run directly through ninja.
Either way it’s probably useful to land this as a separate ninja target in parallel and do the switch-over at a later point just in case something weird breaks. It makes reverts a lot easier.
I think the concerns for that in the past have been around build-time dependencies, and specifically tablegen. Lit is just Python and completely independent from the rest of LLVM, so I think it should be fine even though it lives under llvm/. I don’t think libc has supported building outside of the monorepo, so that shouldn’t be an issue, and lit is easily installable as a python package (well, an old version) if that ends up being an issue.
I’m happy to add lit as an option for LLVM-libc tests. I want to make sure it’s still convenient to open tests with a debugger as well. Currently each test is built as an individual binary which can be run with a debugger. For the libc++ tests I couldn’t find how to do that, so either we need to make it easier or document where the resulting binaries are.
Thanks for the reply, Michael - this proposal specifically is not to change how tests are built, so they remain individual executables. It’s only the harness used to run them. My hope is that this part itself brings enough benefits that it’s worth doing on its own - and brings the possibility of experimenting with other testsuite possibilities (about 50% of CMake’s build generation time and ~80% of my >100MB build.ninja file go away without the test suites). But anything like that is solidly in a future RFC, and only if the trade-offs are right
This would be great. Concretely this would help me to run LLVM libc tests in RISC-V LLVM CI as well without a setup that’s any more special than for the other LLVM tests being run.
To balance iteration time and fidelity, the setup involves cross-compilation and then running unit tests under an appliance VM, and a wrapper to llvm-lit that does this transparently works very successfully for the majority of LLVM tests with no work needed on my part to support additional LLVM subprojects if tests are run through lit. Explicitly, this is a slightly non-standard use case that shouldn’t dictate LLVM design decisions. But at the same time, it’s a useful benefit I would gain if you implement this move to a lit-based harness, so I thought I’d mention it.
@kaladron I looked at the changes, it looks like they are entirely in the cmake build system. In that case, would it be easier to have some cmake flag, let say LIBC_TEST_RUNNER or LIBC_TEST_HARNESS with 2 options at the beginning: LIBC / LIT, and we just update the check-libc target accordingly? We can default the value as LIBC for now and switch to LIT later when it’s ready.
I’m late to the party, but what I do for debugging libc++ tests is:
Run the test using lit --show-all – that builds and runs the test. I can kill the execution of the test if I want. As part of that command, the command-line used to compile the test is printed. That includes the (somewhat obscure) path to the executable.
Run lldb <executable>. That’s it!
Since I likely want debug information, I often add this to my test: // ADDITIONAL_COMPILE_FLAGS: -O0 -g. That works pretty well for me!
I can’t speak for everyone, but I’ve never considered it. I’ve never seen it discussed either.
That being said, there’s a real challenge in doing that: RUN lines in Lit have no flavor, they are all arbitrary commands being run. To run LLDB, you’d have to know which RUN lines pertain to building and other stuff, and which ones pertain to running executables. I can see a greater scheme in which that makes sense and it could deliver additional value (eg to allow running on devices and emulators), but it’s a project in itself.
If you’re interested to invest more time in that general topic, let me know and I can give you some pointers, this is something I’ve given quite a bit of thought over the years. I’m OOO for the next week though so if I don’t reply, I’m not ghosting you.
No worries, I’m also out the next week. =) That said, I’ll probably ping this thread in closer to a month if that’s alright with you. I feel like I’ve gone down a few sidequests on the way to my original goal and I need to make a bit of progress before I find another one.
I can’t speak for everyone, but I’ve never considered it. I’ve never seen it discussed either.
That being said, there’s a real challenge in doing that: RUN lines in Lit have no flavor, they are all arbitrary commands being run. To run LLDB, you’d have to know which RUN lines pertain to building and other stuff, and which ones pertain to running executables. I can see a greater scheme in which that makes sense and it could deliver additional value (eg to allow running on devices and emulators), but it’s a project in itself.
If you’re interested to invest more time in that general topic, let me know and I can give you some pointers, this is something I’ve given quite a bit of thought over the years. I’m OOO for the next week though so if I don’t reply, I’m not ghosting you.
