Description
Most of the day to day tests in LLVM are regression tests executed by Lit, structured as source code or IR to be passed to some binary, rather than test code directly calling the code to be tested. This has many advantages but can make it difficult to predict which code path is executed when the compiler is invoked with a certain test input, especially for edge cases where error handling is involved. The goal of this project is to help developers create good test coverage for their patch and enable reviewers to verify that they have done so. To accomplish this we would like to introduce a tool that can be fed a patch as input, add coverage instrumentation for the affected source files, runs Lit tests, and records which test cases cause each counter to be executed. For each counter we can then report the number of test cases executing the counter, but perhaps more importantly we can also report the number of test cases executing the counter that are also changed in some way by the patch, since a modified line that results in the same test results isn’t properly tested, unless it’s intended to be a non-functional change.
I propose this be implemented in three separate parts:
- Adding an option to llvm-lit to emit the necessary test coverage data, divided per test case (involves setting a unique value to
LLVM_PROFILE_FILEfor eachRUN) - New tool to process the generated coverage data and the relevant git patch, and present the results in a user friendly manner
- Adding a way to non-intrusively (without changing build configurations) enable coverage instrumentation to a build. By building the project normally, touching the files changed by the patch, and rebuilding with
CCC_OVERRIDE_OPTIONSset to add coverage we can lower the overhead of generating and processing coverage of lines not relevant to the patch.
The tooling in step 2 and 3 can be made completely agnostic of the actual test-runner, lowering the threshold for other test harnesses than Lit to implement the same functionality.
If time permits I think adding this as a step in CI would also be helpful for reviewers.
Expected results
Implement a new tool for use by the community. Developers get help finding uncovered edge cases during development, while also avoiding paranoid sprinkling of asserts or logs just to check that the code is actually executed. Reviewers can more easily check which parts of the patch are tested by each test.
Desirable skills
Python for Lit, data processing and diff processing. No compiler experience necessary.
Project size
Small or medium
Difficulty
Simple
Confirmed mentors
@hnrklssn
Happily accepting co-mentors if anyone with coverage experience is interested.