SBCC and emitEmptyMapping

Compared to LLVM’s gcov-compatible coverage method, SBCC’s resource requirements are much bigger. The total size of all .profraw files from my system is almost 1GB while the same system, built for gcov, totals only 150MB of .gcda files. I removed emitEmptyMapping from CoverageMapping.cpp to see what happens, and the SBCC coverage dumps dropped from 1GB to about 26MB. There are also significant gains in runtime memory use and on-disk image size.

I don’t mind losing the zero counts for code that is unused but I want to understand how clang determines what’s used and what isn’t. I traced the decision up to CodeGenPGO::assignRegionCounters in CodeGenPGO.cpp, where implicit functions and some constructor/destructor variants are eliminated but I can’t explain the bulk of it.

Can someone help me understand how clang determines what code is used and what isn’t in this context?

Thanks.

-thom

+ Max

Compared to LLVM’s gcov-compatible coverage method, SBCC’s resource requirements are much bigger. The total size of all .profraw files from my system is almost 1GB while the same system,

Have you tried enabling run-time raw profile merging with "%Nm" (https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program)?

built for gcov, totals only 150MB of .gcda files. I removed emitEmptyMapping from CoverageMapping.cpp to see what happens,

Some engineers working on Chromium tried something similar using an experimental cl::opt (-limited-coverage-experimental) which may have a comparable same effect. I've CC'd Max who may be able to say more about this.

and the SBCC coverage dumps dropped from 1GB to about 26MB. There are also significant gains in runtime memory use and on-disk image size.

I don’t mind losing the zero counts for code that is unused

In this case, I'd suggest trying the cl::opt I mentioned above, with the caveat that it can skew coverage reporting quite a bit.

IMO having some mapping information for unused code is arguably one of the more important use cases for coverage reporting.

but I want to understand how clang determines what’s used and what isn’t.

See AddDeferredUnusedCoverageMapping, ClearUnusedCoverageMapping, and EmitDeferredUnusedCoverageMappings. I'm not the original author so I might not have this totally right, but I think the idea here is that:

- In CodeGenModule::EmitTopLevelDecl, we might not be sure a Decl actually needs to be emitted, so we mark it as deferred for coverage reporting purposes. If we never generate code for it, we'd emit a simple empty coverage mapping.

- If it turns out that a Decl marked as deferred is actually emitted, we clear the deferred bit, because we no longer want an empty mapping.

best,
vedant

I should have mentioned that there is a bug report about the scaling issue:
https://bugs.llvm.org/show_bug.cgi?id=34533

One promising solution is to give the empty coverage mappings names, mark them as linkonce_odr, and store them into a different section. The premise of this approach is that clang will emit identical empty mappings for a function many times, say, if it's an inline function in a header. By marking each of these mappings as linkonce_odr, the linker can unique them in the final binary and save a fair amount of space (O(#functions), instead of O(#functions * #translation-units)).

As I've switched to a different team at work, I don't have the time to work on code coverage bugs at the moment. That said I'd be happy to chat about this further or review patches.

vedant

+ Max

Compared to LLVM’s gcov-compatible coverage method, SBCC’s resource requirements are much bigger. The total size of all .profraw files from my system is almost 1GB while the same system,

Have you tried enabling run-time raw profile merging with "%Nm" (https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#running-the-instrumented-program)?

built for gcov, totals only 150MB of .gcda files. I removed emitEmptyMapping from CoverageMapping.cpp to see what happens,

Some engineers working on Chromium tried something similar using an experimental cl::opt (-limited-coverage-experimental) which may have a comparable same effect. I've CC'd Max who may be able to say more about this.

Er, s/comparable same/comparable/

Yes, “-mllvm -limited-coverage-experimental=true” did help us with Chromium. Things still are crazy large, but smaller than before.

Also, make sure to use “-sparse” option of llvm-profdata tool. It doesn’t emit function records with 0 execution count.

For instance, when generating code coverage for Chromium, we generate 37 GB of .profraw files (1000+ dumps from ~370 binaries), but the resulting .profdata is only ~605 MB. If we merge it without -sparse, it becomes ~878 MB.