[RFC] Integrating SingleByteCoverage with Branch Coverage

I’m proposing changes to integrate SingleByteCoverage with Branch coverage. SingleByteCoverage, introduced in #75425, currently suppresses Branch coverage. I’ve tested a prototype with good results.

Cc: @evodius96 @gulfemsavrun @ellishg

Overview of SingleByteCoverage

SingleByteCoverage uses binary counters set to False (0xFF, initial value) and True (0x00) to save on instruction bytes. It avoids using CounterExpr and suppresses Branch coverage. This works well on manycore builders, where using 64-bit counters is very difficult.

SingleByteCoverage and CounterExpr

The current implementation of SingleByteCoverage avoids CounterExpr. However, CounterExpr::Add is still available because counters can be handled as if their upper limit is 1, but CounterExpr::Subtract is unavailable. Most subtractions happen in BranchRegions, so extra counters are needed for False paths.

Counter expressions from non-SingleByteCoverage can be used by replacing and removing CounterExpr::Subtract.

Proposed Plan

  1. Introduce “CounterPair” for BranchRegions on RegionCounterMap.
  2. Add more counters for False paths.
  3. By enabling CounterExpr, introduce a new option or rewind parts of the current implementation.

Additional Ideas

Reducing Counters more with FalsePath

Example:

C0++; // ParentCnt
if (Cond)
  C1++; // TruePath
else
  C2++; // FalsePath, (C0 - C1)

Replace (C0 - C1) with C2, and apply C0 = (C1 + C2). We can remove C0 if all instances in a function can be replaced. This isn’t always possible.

Selective Counter Activation

BranchRegion flags are exclusive between TruePath and FalsePath. An intrinsic for selective flag setting could help:

Counter[Cond ? TrueIdx : FalseIdx] = true;

If TruePath is replaced with CounterExpr, use:

if (!Cond) Counter[FalseIdx] = true;

This works better with conditional-counter-update mode.

Using MC/DC Test Vectors

As mentioned in 79629, binary counters are equivalent to MC/DC test vectors, which can be used to reduce counter numbers in MC/DC mode.

Thank you.

P.S. I’ll attend the devmtg2024.

1 Like

I will also be at devmtg2024

1 Like

I’m very supportive of this proposal, and thanks for looking into this @chapuni. I’ll be in the Developers’ Meeting as well. We should definitely catch up with all the recent development and the current progress in instrumentation and coverage in general.

1 Like

Not to drift off topic, but if you’re actively working in this area, are there architectural changes we could make to coverage that would make it easy to carry all these benefits over to Rust? IIUC single byte coverage is ABI incompatible with classic coverage, and this presents challenges for Chromium’s use of code coverage.

cc @hansw2000 @ayzhao

I’ve pushed some NFC requests as the preparation.

I’ve pushed almost all requests. I know I have to add more tests.

This is the merged all-in-one branch.

This is a preparation.

They are actual updates.

Please take a look. Feedbacks are welcome.
Thanks.

I added llvm-cov tests for singlebytecoverage. I think my changes are ready for commit.

I’ve created the index into #113115.

Sorry for the long delay. I’d like to land them (1+4 requests in this) hopefully to release 22. All requests under the umbrella are ready for review. (1 preparation and 4 similar codegen changes) @evodius96 @gulfemsavrun @ellishg @nikic

@nikic Could you take a look into CodeGenPGO.cpp changes? I wonder if my changes would be appropriate.

Can I merge them as post-commit review, as a maintainer? I am certain they will affect only “Single byte counters” coverage mode, which is supposed as experimental.

Thanks in advance.

Sorry, I’m not the right person to review this or answer this question. cc @efriedma-quic as clang codegen maintainer and @AaronBallman clang lead maintainer.

My 2c is that if there are no longer active reviewers for coverage (is this the case?) it might be fine to land, but I’d feel very uncomfortable with landing it 3 days before the release branches.

1 Like

Thanks for the comment and sorry to bother you.

I pushed some commits and I guess you noticed them. I think they are NFC or low-impact changes. I know it’d not be good manner but as you see, there have been low activities re. coverage. (guessing vacation or uninterested)
I’d paused them if you gave me an answer in past two days. Finally I poked you with my silent commits. Sorry again.

Re. “Single byte coverge”, I have 4 requests left and they are actual changes for CodeGen. I can postpend them after the release branch and try pushing in the coming release window. They have been really long-running requests.

Sorry, I’m not the right person to review this or answer this question.

I nominated you since I had to modify my changes triggered by #142155 (PGO changes). No need to comment if you suppose my changes are appropriate.

Thanks.

1 Like

There are no active maintainers for code coverage in Clang; we need people who are willing to step up because this area has a lot of unresolved issues (well over 100 by my count). I don’t think we’d like to see coverage support removed from Clang due to it being unmaintained and bitrot, so I will try to see if I can encourage someone to step up. If anyone reading this thread is interested in being a maintainer for code coverage, please speak up (either here or you can reach out to me privately to discuss). Currently, code coverage in Clang is handled by falling back to the codegen code owners, but they’ve already got a lot on their plate in terms of workload.

+1

2 Likes

Let me defend. Alan @evodius96, another maintainer, has been working as a reviewer while I was away due to a few personal reasons for months.

That said, my pull requests haven’t been reviewed for months. Some of them were aiming to the release 20. I could guess Alan would be in the vacation.

I know “Code coverage functionality” would less interest anyone. In fact, I was not in the past till my current position.

1 Like

I apologize for the silence on your many other issues; just juggling priorities a bit.

I can certainly assist with reviews for code coverage Clang since I have contributed to this in the past as well, and I know teams rely on it. With this particular project, I really would like to include folks who’ve worked on SingleByteCoverage, especially @gulfemsavrun, to review it.

2 Likes

Thank you for the offer to help with reviews in Clang! Would you be comfortable if I nominated you for official maintainership so that we’ve got a community point of contact for help in the area? (Please don’t feel pressured; “no” is a perfectly acceptable answer.)

@AaronBallman
I was nominated by @nikic in the past. I asked nominating @evodius96 as another maintainer then. I think two maintainers will be sufficient here.

@gulfemsavrun
I didn’t rush you much but I have been waiting for you to review. You are the originator of “Single byte”.

Or can I ask reviewing to clangCodeGen owner, @efriedma-quic ?

1 Like

Definitely, thank you! I’m in the process of doing a refresh for maintainers for the Clang Area Team, so I’ve put up a PR nominating you both and add you as reviewers.

1 Like

Oh excuse me, I didn’t distinguish “Clang” area re. coverage.

I think this is one of those areas where it cuts across multiple projects (Clang, LLVM, llvm-cov), so if you’re not comfortable maintaining the Clang bits, let me know and I can retract the nomination. Same goes for @evodius96 – I’m not trying to pressure folks into work they’re not interested in. But on the Clang side, we do run into occasional coverage issues and currently those fall back to the codegen maintainers who already have a very heavy review workload.

1 Like

I’m going to review the changes for single-byte counters. Could you clarify if the integration of single-byte counters and branch coverage is currently working end-to-end? Specifically, are the pending PRs sufficient to enable this, or is additional implementation still required?

@gulfemsavrun Thanks for taking a look!

You can find my four unmerged requests (and squashed view) in #113115 .

I suppose they are sufficient. Each change enables branch functionalities (see updated testcases). W/o my changes, the behavior should be same as the current implementation.