Profiling report with branch coverage for templates is misleading

Hi Everyone,

I am creating a branch coverage report using the tool “genhtml” of code instrumented with clang. The branch coverage for template code is low, especially if many instances with different template parameters exist.

The report is generated using “clang” and “llvm-cov” with the following script:

#!/bin/bash
set -e

clang++ -O0 -fcoverage-mapping -fprofile-instr-generate -DWITH_USAGE $@ template_coverage.cpp

./a.out
llvm-profdata merge -output=default.profdata default.profraw
llvm-cov export ./a.out -instr-profile=default.profdata . -format lcov > "lcov.info"
genhtml --function-coverage --branch-coverage --show-details -o genhtml "lcov.info"

I have reduced the case into an example in file “template_coverage.cpp”:

#include <iostream>
#include <optional>
#include <string>
#include <vector>

template<typename T, typename F>
auto orElse(const std::optional<T>& opt, F elseFunc) {
	if (opt.has_value()) {
		return opt.value();
	}
	return elseFunc();
}

// Example of a test that runs all branches of function orElse
static std::string testCoverageWithString() {
	std::optional<std::string> opt1;
	std::optional<std::string> opt2 = "valid";
	auto elseFunc = []() {
		return std::string("alternative");
	};
	auto v1 = orElse(opt1, elseFunc);
	auto v2 = orElse(opt2, elseFunc);
	return v1 + "," + v2;
}

#ifdef WITH_USAGE
// Example of code that makes use of function orElse, where not all branches
// have to be tested again
static std::vector<int> usageWithVector() {
	std::optional<std::vector<int>> opt1;
	return orElse(opt1, []() {
		return std::vector<int>();
	});
}
#endif

int main() {
	std::cout << testCoverageWithString() << std::endl;

#ifdef WITH_USAGE
	usageWithVector();
#endif
	return 0;
}

It seems that the coverage report handles every instantiation of template function “orElse” independently. This leads to an incomplete branch coverage for this function.

Here is an extract of the file “lcov.info”:

SF:/home/stma6092/Documents/LLVM/template_coverage.cpp
...
FN:7,template_coverage.cpp:_Z6orElseINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEZL22testCoverageWithStringvE3$_0EDaRKSt8optionalIT_ET0_
...
FN:7,template_coverage.cpp:_Z6orElseISt6vectorIiSaIiEEZL15usageWithVectorvE3$_0EDaRKSt8optionalIT_ET0_
...
DA:7,3
DA:8,3
DA:9,1
DA:10,1
DA:11,2
DA:12,3
...
BRDA:8,0,0,1
BRDA:8,0,1,1
BRDA:8,1,2,0
BRDA:8,1,3,1
BRF:2
BRH:2
...
end_of_record

There are four BRDA entries for line 8 with two different block numbers. Is there any option in “llvm-cov” to reduce the number of considered blocks to a union of all instantiated templates?

If not, is there any way to identify the different template instances in the BRDA entries to apply some kind of post-processing after “llvm-cov export”?

Hi @stma2024 Without exporting to LCOV, can you dump the output from llvm-cov using text output (llvm-cov show --format=text ...)? What does that look like? I’d like to ascertain whether there is a problem with the data visualization itself as opposed to LCOV.

Thanks!

Hi @evodius96,

here is the output of “llvm-cov show”. I used the following command:

llvm-cov show --format=text --show-branches=count ./a.out -instr-profile=default.profdata >"lcov.text"

The file content of “lcov.text” is:

    1|       |#include <iostream>
    2|       |#include <optional>
    3|       |#include <string>
    4|       |#include <vector>
    5|       |
    6|       |template<typename T, typename F>
    7|      3|auto orElse(const std::optional<T>& opt, F elseFunc) {
    8|      3|	if (opt.has_value()) {
  ------------------
  |  Branch (8:6): [True: 1, False: 1]
  |  Branch (8:6): [True: 0, False: 1]
  ------------------
    9|      1|		return opt.value();
   10|      1|	}
   11|      2|	return elseFunc();
   12|      3|}
  ------------------
  | template_coverage.cpp:_Z6orElseINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEZL22testCoverageWithStringvE3$_0EDaRKSt8optionalIT_ET0_:
  |    7|      2|auto orElse(const std::optional<T>& opt, F elseFunc) {
  |    8|      2|	if (opt.has_value()) {
  |  ------------------
  |  |  Branch (8:6): [True: 1, False: 1]
  |  ------------------
  |    9|      1|		return opt.value();
  |   10|      1|	}
  |   11|      1|	return elseFunc();
  |   12|      2|}
  ------------------
  | template_coverage.cpp:_Z6orElseISt6vectorIiSaIiEEZL15usageWithVectorvE3$_0EDaRKSt8optionalIT_ET0_:
  |    7|      1|auto orElse(const std::optional<T>& opt, F elseFunc) {
  |    8|      1|	if (opt.has_value()) {
  |  ------------------
  |  |  Branch (8:6): [True: 0, False: 1]
  |  ------------------
  |    9|      0|		return opt.value();
  |   10|      0|	}
  |   11|      1|	return elseFunc();
  |   12|      1|}
  ------------------
   13|       |
   14|      1|static std::string testCoverageWithString() {
   15|      1|	std::optional<std::string> opt1;
   16|      1|	std::optional<std::string> opt2 = "valid";
   17|      1|	auto elseFunc = []() {
   18|      1|		return std::string("alternative");
   19|      1|	};
   20|      1|	auto v1 = orElse(opt1, elseFunc);
   21|      1|	auto v2 = orElse(opt2, elseFunc);
   22|      1|	return v1 + "," + v2;
   23|      1|}
   24|       |
   25|       |#ifdef WITH_USAGE
   26|      1|static std::vector<int> usageWithVector() {
   27|      1|	std::optional<std::vector<int>> opt1;
   28|      1|	return orElse(opt1, []() {
   29|      1|		return std::vector<int>();
   30|      1|	});
   31|      1|}
   32|       |#endif
   33|       |
   34|      1|int main() {
   35|      1|	std::cout << testCoverageWithString() << std::endl;
   36|       |
   37|      1|#ifdef WITH_USAGE
   38|      1|	usageWithVector();
   39|      1|#endif
   40|      1|	return 0;
   41|      1|}

Thank you! Sorry for the delay. I dug into this a little bit and I have the following observations:

1.) If you dump the summary using llvm-cov show without exporting to LCOV, the summary does do the union of function instantiations for branches, with the above showing 100% branch coverage for the function. This is the behavior I think you expect to see for LCOV. Also, if you show the report with individual functions, it will show 50% coverage for one of the instantiations and 100% for the other, which is the expected behavior, and is similar to the source-level view also shown above.

2.) When exporting to LCOV, the BRF/BRH information is derived from the same summary, and as you can see, it correctly shows two branches, both taken, which would yield 100% branch coverage. However, the problem is that the BRDA information is calculated per-function (and therefore per-function-instantiation) and simply enumerates branches on the same line without considering whether the information is from an instantiation. When it sees BRDA information, LCOV is basing the summary on that information rather than on BRF/BRH.

Although I can’t explain why LCOV uses BRDA information for the summary rather than BRF/BRH, I suspect that there is a bug in the llvm-cov exporter where the BRDA data should not be enumerated across function template instantiations and instead the max taken. So instead of this:

BRDA:8,0,0,1
BRDA:8,0,1,1
BRDA:8,1,2,0
BRDA:8,1,3,1

we should see this:

BRDA:8,0,0,1
BRDA:8,0,1,1
BRDA:8,0,0,0
BRDA:8,0,1,1

I think LCOV would then show the right summary.

It may be possible to workaround this using tiarmcov export --summary-only, to exclude the BRDA information, forcing LCOV should show you the right summary. But for some reason, genhtml is not processing this information correctly for me, so I can’t confirm that.

-Alan

Hi Alan,

you are right. The implementation of “llvm-cov export” does not seem to consider the template instantiations at all, as opposed to “llvm-cov show”.

I had a deeper look into the source code and the implementation of “export” mode in function https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/tools/llvm-cov/CoverageExporterLcov.cpp#L148 is based on https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/tools/llvm-cov/CoverageExporterLcov.cpp#L186.

The “show” mode with option “–show-instantiations” instead does more and is based on https://github.com/llvm/llvm-project/blob/llvmorg-20-init/llvm/tools/llvm-cov/CodeCoverage.cpp#L424.

Using this approach should give the export mode the necessary information to output the “BRDA” statements in the way you suggested (by using the same block number).

Do I have to create a bug ticket at https://github.com/llvm/llvm-project/issues or is that a request for a new feature?

-Stefan

Hi Stefan,

I would treat this as a bug in CoverageExporterLCov.cpp for how branch information is displayed across template instantiations. If you like, you can go ahead and fix it!

Did you try using tiarmcov export --summary-only as a workaround?

-Alan

Hi Alan,

I have created a new issue at Profiling report with branch coverage for templates is misleading · Issue #111743 · llvm/llvm-project · GitHub and started to solve it.
Currently, I have chosen to fix it with minimal changes - just by combining branches at the same location into one. That’s the behavior I have seen using “gcc” and “gcovr --lcov”.

Another approach would be to keep the branches of each template instance by using the same block and branch number for equal branch locations. The tool “genhtml” seems to handle such cases quite well.

Unfortunately, I didn’t find the time to install and use the tool “tiarmcov”.

-Stefan

Sorry for the confusion – I meant llvm-cov export --summary-only. tiarmcov is our rename of this tool for our downstream Arm toolchain.