Guidance on Issues with Code Coverage Accuracy

[justincady]

Hi,

There are some code coverage accuracy issues affecting my team:

• [coverage] incorrect coverage when non-throwing code exists within a try block · Issue #138871 · llvm/llvm-project · GitHub
  • I suspect this is related to exception handling code generation.
• LLVM Source-Based coverage reports missing coverage on closing braces of case blocks in switch statements · Issue #64956 · llvm/llvm-project · GitHub
  • I suspect this is a region mapping issue.
• [coverage] nested macro with unused branch leads to incorrect coverage · Issue #140893 · llvm/llvm-project · GitHub
  • I suspect this is a region mapping issue.

I am not asking for someone to pick these up and fix them, though if you want to do that I would be thankful!

@evodius96 suggested a Discourse post to get some visibility.

If anyone is able to provide some guidance (or wants to collaborate) on any of the above issues, I’d appreciate it.

Thanks!

cc @evodius96 @chapuni @ZequanWu @vedantk

[justincady]

Hi,

Following up, some coverage accuracy issues have been resolved:

• [coverage] __noreturn__ attribute leads to incorrect coverage report · Issue #124296 · llvm/llvm-project · GitHub
• [coverage] incorrect coverage when a do-while contains break or continue statements · Issue #139122 · llvm/llvm-project · GitHub

Here are the remaining known issues affecting my team (some newly filed since my original post):

• [coverage] incorrect coverage when non-throwing code exists within a try block · Issue #138871 · llvm/llvm-project · GitHub
• [coverage] nested macro with unused branch leads to incorrect coverage · Issue #140893 · llvm/llvm-project · GitHub
• [coverage] macro with return statement leads to incorrect coverage · Issue #142203 · llvm/llvm-project · GitHub
• [coverage] incorrect coverage for lines following conditional terminating statements · Issue #144416 · llvm/llvm-project · GitHub
• LLVM Source-Based coverage reports missing coverage on closing braces of case blocks in switch statements · Issue #64956 · llvm/llvm-project · GitHub

If anyone with relevant expertise is able to provide guidance on any of these, I would appreciate the help.

I’d also appreciate any big picture perspectives. I wonder if others:

1. Never observe coverage accuracy issues
2. Rarely observe coverage accuracy issues but find overall accuracy to be “good enough”
3. Frequently observe coverage accuracy issues but resolve them in another way

Thanks!

[evodius96]

Thank you for looking at these, @justincady! And I appreciate the summary of remaining issues you’ve seen.

@justincady

I’d also appreciate any big picture perspectives. I wonder if others:

1. Never observe coverage accuracy issues
2. Rarely observe coverage accuracy issues but find overall accuracy to be “good enough”
3. Frequently observe coverage accuracy issues but resolve them in another way

Our downstream compilers are used in the functional safety embedded space. In those code bases, the code tends to be simple. Mostly C, and very rarely or never are code statements included in macros. Branches tend to be 2-3 conditions max. So most of the issues you point out are never observed (1.).

Most important for me would be addressing the issues that directly impact the overall coverage metric in the coverage report that shows code coverage percentage. That metric is what our users generally look to first. If it’s not 100%, then the source-code view is useful to identify uncovered lines. If that coverage metric is accurate, then the source-code view is “good enough” to figure things out. However, I’m concerned about when the metric is not accurate, or what it might be hiding, such as when it might show 100% coverage when it isn’t in the issues you’ve seen.

If I sort your remaining issues according to the use cases I encounter most, I would prioritize them this way:

Most important (impacts overall coverage metric?)

• LLVM Source-Based coverage reports missing coverage on closing braces of case blocks in switch statements · Issue #64956 · llvm/llvm-project · GitHub
• [coverage] incorrect coverage for lines following conditional terminating statements · Issue #144416 · llvm/llvm-project · GitHub

Less important (stmts and branches in macros):

• [coverage] macro with return statement leads to incorrect coverage · Issue #142203 · llvm/llvm-project · GitHub
• [coverage] nested macro with unused branch leads to incorrect coverage · Issue #140893 · llvm/llvm-project · GitHub

Least important (try/catch):

• [coverage] incorrect coverage when non-throwing code exists within a try block · Issue #138871 · llvm/llvm-project · GitHub

I might be able to get some from my team to look at at least one of the “most important” ones. I’d like to know what you or others think.