[Coverage] Patch based test coverage for quick test feedback


Most of the day to day tests in LLVM are regression tests executed by Lit, structured as source code or IR to be passed to some binary, rather than test code directly calling the code to be tested. This has many advantages but can make it difficult to predict which code path is executed when the compiler is invoked with a certain test input, especially for edge cases where error handling is involved. The goal of this project is to help developers create good test coverage for their patch and enable reviewers to verify that they have done so. To accomplish this we would like to introduce a tool that can be fed a patch as input, add coverage instrumentation for the affected source files, runs Lit tests, and records which test cases cause each counter to be executed. For each counter we can then report the number of test cases executing the counter, but perhaps more importantly we can also report the number of test cases executing the counter that are also changed in some way by the patch, since a modified line that results in the same test results isn’t properly tested, unless it’s intended to be a non-functional change.

I propose this be implemented in three separate parts:

  1. Adding an option to llvm-lit to emit the necessary test coverage data, divided per test case (involves setting a unique value to LLVM_PROFILE_FILE for each RUN)
  2. New tool to process the generated coverage data and the relevant git patch, and present the results in a user friendly manner
  3. Adding a way to non-intrusively (without changing build configurations) enable coverage instrumentation to a build. By building the project normally, touching the files changed by the patch, and rebuilding with CCC_OVERRIDE_OPTIONS set to add coverage we can lower the overhead of generating and processing coverage of lines not relevant to the patch.

The tooling in step 2 and 3 can be made completely agnostic of the actual test-runner, lowering the threshold for other test harnesses than Lit to implement the same functionality.
If time permits I think adding this as a step in CI would also be helpful for reviewers.

Expected results

Implement a new tool for use by the community. Developers get help finding uncovered edge cases during development, while also avoiding paranoid sprinkling of asserts or logs just to check that the code is actually executed. Reviewers can more easily check which parts of the patch are tested by each test.

Desirable skills

Python for Lit, data processing and diff processing. No compiler experience necessary.

Project size

Small or medium



Confirmed mentors

Happily accepting co-mentors if anyone with coverage experience is interested.


In case you can get this to support clang-format-like git integration, that would be awesome! git check-llvm HEAD^ maybe?