Coverage from multiple Test Executables

Clang Frontend Using Clang
1

Hi, at the moment I am trying to generate a code coverage report which should combine the coverage of multiple test executables, i.e. I have a static library A and e.g. two executables linking agains that library and executing tests. What I do today is:

  • Building the library and tests with -fprofile-instr-generate and -fcoverage-mapping (and also -fno-inline -fno-exceptions).
  • Then running both test executables TestA and TestB with environment variable LLVM_PROFILE_FILE set to different files.
  • Then combine the profraw files created using llvm-profdata merge -o merge.profdata …(all profraw files from above).

That gives me a single profdata file I now want to process using llvm-cov command. I try to run llvm-cov show --instr-profile=merge.profdata -object TestA TestB.

That generates a coverage report and looks also good overall. However, I run into an issue for a specific functions (actually in a header file) which are not being covered in the report, although there are tests calling the function. The very interesting thing here is, in case I reorder the llvm-cov call with -object TestB TestA it shows coverage as expected.

Now my question: Is this a use-case that should work? Can you explain why the order of the executables in the llvm-cov makes a difference (and should it be like this?). Do you have any advice about a correct approach?

I will try to produce a minimal example of this behavior.

1 Like
2

Here you find a minimal example including the commands I used and the generated output files Coverage from multiple Test Executables · GitHub

I also noticed that refactoring the method tested here also resolves the issue somehow. So something must be special here.

3

I also tried to debug a little bit the llvm-cov tool and found inside CoverageMapping.cpp two interesting places. The following if becomes true and thus the proper coverage of the function is skipped when the test1 is the second one loaded.

image

4

However, I would have expected the if shoto hit for the test2 where the function is actually unused. But Counts[0] is actually == 0 (while the other two conditions are true).

image

The 0 in the Counts comes from the instrprof_error::unknown_function error above which sets 0 to Counts.

5

Maybe this is the same root cause as in here [llvm-cov] Hash mismatches originating from class methods implemented in header files - #10 by Aleksa_Markovic?

6

Greetings Christoph,

In a few words, you need to specify --object for each individual object file you’re passing to llvm-cov:
llvm-cov show --instr-profile=merge.profdata --object TestA --object TestB

I would call this a design flaw of llvm-cov.

A warm suggestion is to use llvm-cov only to export data to LCOV tracefile format and then continue all processing and report generation using lcov. llvm-cov cannot reliably do merging of data from multiple sources.

Good luck,
Aleksa

7

Hi Aleksa,

thank you very much for your feedback. Are you sure using --object multiple times solves the issue? I wondered that it solves it for me too, but then I checked the source code of llvm-cov and had no explanation why. Using --dump-collected-objects gave me the hint:

–object TestA --object TestB results in TestA processed first, then TestB
–object TestA TestB switches the order, so it is equivalent to --object TestB --object TestA

So I think the order here is what fixed the issue for me (just due to the internal processing in llvm-cov where I am not proficient enough right now to judge on it. But I think there is no “correct” order, because it depends on the function we look at what’s the correct order.

Isn’t the actual merging happening in llvm-profdata where merge the two profraw to a single profdata file?

8

Hi @Aleksa_Markovic , I also tried experimenting with merging the exportet lcov data but that also have a major flow.

In case a method is not generated (since in a header and not called) the lcov files contain a “0 hits” entry for each line of the method, even for lines which do not contain any executable code. However, when I look into the coverage when the method is called, these empty lines are not asssociated to coverage.

When now merging, we cannot decide what to do with these lines, we would need to apply some kind of magic to only take lines which are in all lcovs, but that would again eliminate totally uncovered function.

So I think this entire thing cannot be workarounded at the moment :frowning:

1 Like
9

Can confirm that it shows line count for the function in header file after removing this condition.

/tmp/cov/ToTest.h:
    1|       |
    2|       |#include <cstdint>
    3|       |
    4|       |namespace test
    5|       |{
    6|       |
    7|       |const uint8_t NoError = 16U;
    8|       |const uint8_t Unknown = 17U;
    9|       |const uint8_t ErrorCount = 16U;
   10|       |
   11|       |struct Test
   12|       |{
   13|       |    bool isError[ErrorCount]{false};
   14|       |    bool isUnknown = false;
   15|       |
   16|      1|    Test() = default;
   17|       |
   18|       |    uint8_t getToTest() const
   19|      1|    {
   20|     17|        for (uint8_t i = 0U; i < ErrorCount; i++)
   21|     16|        {
   22|     16|            if (isError[i])
   23|      0|            {
   24|      0|                return i;
   25|      0|            }
   26|     16|        }
   27|       |
   28|      1|        if (isUnknown)
   29|      0|        {
   30|      0|            return Unknown;
   31|      0|        }
   32|       |
   33|      1|        return NoError;
   34|      1|    }
  ------------------
  | Unexecuted instantiation: _ZNK4test4Test9getToTestEv
  ------------------
  | _ZNK4test4Test9getToTestEv:
  |   19|      1|    {
  |   20|     17|        for (uint8_t i = 0U; i < ErrorCount; i++)
  |   21|     16|        {
  |   22|     16|            if (isError[i])
  |   23|      0|            {
  |   24|      0|                return i;
  |   25|      0|            }
  |   26|     16|        }
  |   27|       |
  |   28|      1|        if (isUnknown)
  |   29|      0|        {
  |   30|      0|            return Unknown;
  |   31|      0|        }
  |   32|       |
  |   33|      1|        return NoError;
  |   34|      1|    }
  ------------------
   35|       |};
   36|       |
   37|       |}

But it also shows an error “Unexecuted instantiation: _ZNK4test4Test9getToTestEv” with a subview of the function body.

Another thing I noticed is that the main functions in in test1.exe and test2.exe show execution count 2 instead of 1, which is incorrect. My guess is that it double counts main function in both binaries:

/tmp/cov/test1.cpp:
    1|       |#include <iostream>
    2|       |
    3|       |#include "ToTest.h"
    4|       |
    5|       |int main(int argc, char **argv)
    6|      2|{
    7|      2|    std::cout << "executed test1" << std::endl;
    8|      2|    test::Test t;
    9|      2|    std::cout << (int)t.getToTest() << std::endl;
   10|      2|    return 0;
   11|      2|}

/tmp/cov/test2.cpp:
    1|       |#include <iostream>
    2|       |
    3|       |#include "ToTest.h"
    4|       |
    5|       |int main(int argc, char **argv)
    6|      2|{
    7|      2|    std::cout << "executed test2" << std::endl;
    8|      2|    return 0;
    9|      2|}
11

Okay, I also had some further look today to try to understand what’s going on and I think we have some more issues. Let me try to summarize my observation:

When the method in the header file is not called, it is not emitted by the compiler. However, the coverage information in the binary still contains the function with a hash of zero. If we now consider two test test executables, of which on calls the method and the other one not, execute both and merge the profraw files into a single profdata, we get the following situation:

  • test1: does call the method and the coverage information contains the “correct” hash != 0
  • test2: does not call the method and the coverage information contains the hash == 0
  • profdata: contains counters from the execution of test1 with the “correct” hash

The llvm-cov tool now reads the two test binaries in the order as provided on the commandline. It now sees the not-emitted function with hash zero and stores it internally (and assuming all counters to be zero, see llvm-project/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp at dcf0160bd61d150e7b94067fcd991b466a361b08 · llvm/llvm-project · GitHub). When later the acutally emitted function with hash != 0 is processed, it is ignored because the implementation thinks that the method is already processed. If we now change the order of the binaries on the command line, by accident, the one which has the hash != 0 is processed first, resulting in the expected report.

To make the behavior even harder to understand, there are situations in which it still works although the binaries are processed in the “wrong” order. This is, because the implementation in getInstrProfRecord (llvm-project/llvm/lib/ProfileData/InstrProfReader.cpp at dcf0160bd61d150e7b94067fcd991b466a361b08 · llvm/llvm-project · GitHub) checks for something called CSFlagInHash (not sure what exactly that is) in the hash. For the hash == 0 this is always false, but for the data in the profdata it can be the one or the other (depending on what the compiler generated, feels random from user perspective). In case the CSFlagInHash in the profdata is now false, the function with hash == 0 is reported as a hash_mismatch, causing it being ignored (llvm-project/llvm/lib/ProfileData/Coverage/CoverageMapping.cpp at main · llvm/llvm-project · GitHub).

12

Great analysis. Maybe coverage should treat functions with hash being 0 specially such that they can be merged to the functions with the same names but have non-zero hash.

13

Looks like an oversight on this commit: [Coverage] Ignore 'unused' functions with non-zero execution counts · llvm/llvm-project@381e9d2 · GitHub.

I tried to change the condition from Counts[0] > 0 to Counts[0] == 0. It makes the example you provided to work but failed a test in compiler-rt/test/profile/instrprof-merging.cpp (specifically line 20). I’ll take a look on that a bit more.

1 Like
14

Sent a fix at: [Coverage] Ignore unused functions if the count is 0. by ZequanWu · Pull Request #107661 · llvm/llvm-project · GitHub.

15

Your guess is correct and it’s due to main() having the exact same symbol name and hash:

$ llvm-profdata show test1.profraw --all-functions
Counters:
  main:
    Hash: 0x0000000000000018
    Counters: 1
    Function count: 1
# ...
$ llvm-profdata show test2.profraw --all-functions
Counters:
  main:
    Hash: 0x0000000000000018
    Counters: 1
    Function count: 1

Even though this is a minimal example, I can imagine there will be cases where small, branch-less functions merge like this even though they shouldn’t. We should consider changing the way hashes are generated.
Cc: @chapuni

Best regards,
Aleksa

16

To further clarify my statement on unreliable merging:
Yes, but the merge criteria is a matching hash and symbol name. There is no information about the source file, as this mapping exists only in the object file containing the symbols. In ZequanWu’s analysis, we see that two different main()s merge into one because their hashes and names matched.

After better understanding what you’ve presented here, I think the only way to get consistent results is:

  • not to do any merging, just convert data from a single profraw to a single profdata
  • similarly, do not pass more than one object file to llvm-cov. You can perform matching of which profraw originate from which object file using build IDs (llvm-profdata show --binary-ids , introduce with clang++ -Wl,--build-id)
  • export the data to LCOV tracefile format and continue all manipulations with lcov.

This is all from experience moving from gcov-compatible coverage to Source-based for a 5 MLOC codebase. I really wish the tools Source-based have these issues resolved, as it’s very hard to get trustworthy and consistent results with so many bugs and workarounds.