Code coverage report for Clang tests and LLVM tests showing same coverage values

LLVM Project
,
1

Hello,

I am exploring to improve code coverage in LLVM/Clang and locally replicating:
https://lab.llvm.org/coverage/coverage-reports/coverage/Users/buildslave/jenkins/workspace/coverage/llvm-project/index.html

Commands used:
mkdir llvm-project/build
cd llvm-project/build
CC=‘clang’ CXX=‘clang++’ cmake -DLLVM_ENABLE_PROJECTS=“clang;llvm;polly;lld” -DCMAKE_BUILD_TYPE=Release -DLLVM_USE_SANITIZER=Address -DLLVM_ENABLE_ASSERTIONS=On -DLLVM_BUILD_INSTRUMENTED_COVERAGE=On -G “Unix Makefiles” …/llvm
make

Test 1: Execute clang test and generate code coverage report in my-coverage-output-dir_clang

make check-clang
python /llvm/utils/prepare-code-coverage-artifact.py --preserve-profiles -C build $(which llvm-profdata) $(which llvm-cov) build/profiles my-coverage-output-dir_clang build/bin/clang

Totals: 59.32% (101503/171118) 82.34% (5325178/6467207) 60.92% (1453672/2386306) 54.35% (977326/1798080)

Test 2: Execute llvm test and generate code coverage report in my-coverage-output-dir_llvm

make check-llvm
python /llvm/utils/prepare-code-coverage-artifact.py --preserve-profiles -C build $(which llvm-profdata) $(which llvm-cov) build/profiles my-coverage-output-dir_llvm build/bin/clang

Totals: 59.33% (101517/171111) 82.35% (5325561/6467190) 60.93% (1454028/2386299) 54.38% (977823/1798080)

After we execute both tests and generate code coverage one-after-another, we see almost same coverage values for Function coverage, Line coverage, Region Coverage and Branch Coverage.

Any suggestions how to fix this?

Thanks and Regards,
Vrukesh

2

Been a while since I did any of this, so broad suggestions:

  • Are you clearing out the previous run’s coverage files between each run?
    (or using separate build directories)
  • Why are you passing build/bin/clang in both cases? (I don’t know that that’s wrong to do, it just stands out to me. Could be correct though.)
  • Have you tried to prove that the second result is what you think it is? For example, could you delete all the tests for some tool or target and see if the totals change. If not, then something needs fixing.
  • Have you looked at the coverage report to see what files are covered in each? If both report a lot of stuff in clang, that’s odd because check-llvm wouldn’t be running clang.

Please link to any documentation you have followed to make any assumptions, so we can check if there’s anything misleading.

Key missing piece here is coverage, of what exactly? Of clang, of the source files in llvm-project/ ?

For instance, if I ran check-llvm, I wouldn’t expect to see any coverage of llvm-project/bolt. So that’s one way to check if the setup works.

3

Hi David,
Thanks for detailed queries and deep-diving ! Please find the responses below.

Are you clearing out the previous run’s coverage files between each run?
(or using separate build directories)
Vrukesh: Intention is to consolidate the code coverage reports of “make check-clang” and “make check-llvm”. So, deliberately, we did not use different “build” directories for these commands. Do you have any suggestion here?

Why are you passing build/bin/clang in both cases? (I don’t know that that’s wrong to do, it just stands out to me. Could be correct though.)
Vrukesh: As per python script, this is “path to an instrumented binary”. Also, I referred Coverage report of the `check-clang-analysis` target for this.

Have you tried to prove that the second result is what you think it is? For example, could you delete all the tests for some tool or target and see if the totals change. If not, then something needs fixing.
Vrukesh: Since “make” is taking lot of time with instrumented code, am dreading doing this. Still, how will I consolidate both the results?

Have you looked at the coverage report to see what files are covered in each? If both report a lot of stuff in clang, that’s odd because check-llvm wouldn’t be running clang.
Vrukesh: This list also seems same.

Regards,
Vrukesh

4

Sure, that makes some sense to save build time. I expect you will need to at least clear out the coverage data between checks though because…

I think that means one or all of them. So you may have a problem here.

check-clang will run clang and llvm tools.
check-llvm will run llvm tools.

I’m not sure you will be able to separate what came from check-clang and what came from check-llvm. Though you could remove all the coverage data files in between the check runs.

My guess is that the check-llvm step will want llvm-mc, llvm-as, etc. as the instrumented binaries. The script would be aggregating coverage data for the binary (binaries?) it’s told about.

    parser.add_argument(
        "binaries",
        metavar="B",
        type=str,
        nargs="*",
        help="Path to an instrumented binary",
    )

The help text is misleading. Should be “Path to at least one instrumented binary.”. I’m pretty sure the intent is you pass all relevant binaries here.

Does that make sense now? When you run the script with clang, it’s only going to find coverage generated by running clang.

Well, it is the blunt approach so you don’t need to do this if you can prove it another way. Which I think you can.

My basic idea was if I deleted a bunch of test cases in llvm/test/Codegen/AArch64 and the coverage report didn’t change at all - you know you’re doing something incorrect.

It’s like writing tests that fail to begin with, just so you can be sure they actually do run (a mistake I’ve made many times).

Yeah I think you need to be passing different instrumented binaries each time. If they were in separate dirs, bin/* would be the list, but you’ll need to be careful here what you include.

Or as I say, remove the coverage data files between check-clang and check-llvm.

Maybe consider the reasons for wanting a separate report for check-llvm and check-clang in the first place.

For instance, my assumption is there will be files that aren’t covered well by tests that run clang. However they will be by tests that run other llvm tools. So it could make for a confusing set of reports.

But I don’t have the context, just thinking out loud. Take that for what you will.

6

Hi David,
Thanks for the inputs. As suggested, I added all the valid binaries from build/bin directory and now, it generates different folders corresponding for each binary.
I will analyze this now, and get back in case of further queries.
Adding the sample command used for future reference.

python /home/vrupan01/llvm-project/llvm/utils/prepare-code-coverage-artifact.py --preserve-profiles -C build $(which llvm-profdata) $(which llvm-cov) build/profiles my-coverage-output-dir_llvm_clang_12122023 build/bin/llvm-exegesis build/bin/llvm-reduce build/bin/clang-repl build/bin/llvm-cov build/bin/llvm-undname build/bin/llvm-rtdyld build/bin/sanstats build/bin/llvm-dwarfdump build/bin/llvm-as build/bin/not build/bin/c-index-test build/bin/llvm-jitlink-executor build/bin/clang-refactor build/bin/llvm-cxxmap build/bin/llvm-cfi-verify build/bin/llvm-dwp build/bin/llvm-libtool-darwin build/bin/clang-extdef-mapping build/bin/llvm-lto build/bin/llvm-remarkutil build/bin/llvm-gsymutil build/bin/llvm-xray build/bin/sancov build/bin/llvm-rc build/bin/llvm-ml build/bin/clang-linker-wrapper build/bin/llvm-modextract build/bin/llvm-mt build/bin/diagtool build/bin/llvm-sim build/bin/llvm-objcopy build/bin/llvm-dlang-demangle-fuzzer build/bin/dsymutil build/bin/llvm-extract build/bin/llvm-debuginfod-find build/bin/obj2yaml build/bin/llvm-symbolizer build/bin/FileCheck build/bin/llvm-cxxfilt build/bin/count build/bin/clang-offload-packager build/bin/llvm-cat build/bin/llvm-mca build/bin/llc build/bin/llvm-min-tblgen build/bin/apinotes-test build/bin/clang-offload-bundler build/bin/llvm-jitlink build/bin/clang-rename build/bin/llvm-mc build/bin/llvm-lto2 build/bin/llvm-profdata build/bin/llvm-readobj build/bin/llvm-link build/bin/llvm-strings build/bin/lli-child-target build/bin/llvm-diff build/bin/clang-diff build/bin/yaml2obj build/bin/llvm-nm build/bin/amdgpu-arch build/bin/llvm-tblgen build/bin/llvm-pdbutil build/bin/llvm-size build/bin/llvm-PerfectShuffle build/bin/llvm-itanium-demangle-fuzzer build/bin/clang-ast-dump build/bin/llvm-dis build/bin/llvm-yaml-numeric-parser-fuzzer build/bin/llvm-special-case-list-fuzzer build/bin/llvm-lipo build/bin/nvptx-arch build/bin/clang-scan-deps build/bin/opt build/bin/clang-import-test build/bin/llvm-tli-checker build/bin/llvm-bcanalyzer build/bin/llvm-opt-fuzzer build/bin/clang-18 build/bin/llvm-dwarfutil build/bin/llvm-ifs build/bin/llvm-opt-report build/bin/llvm-readtapi build/bin/llvm-config build/bin/bugpoint build/bin/llvm-microsoft-demangle-fuzzer build/bin/clang-fuzzer-dictionary build/bin/verify-uselistorder build/bin/arcmt-test build/bin/clang-format build/bin/llvm-cvtres build/bin/llvm-stress build/bin/llvm-cxxdump build/bin/yaml-bench build/bin/llvm-ar build/bin/llvm-profgen build/bin/llvm-objdump build/bin/lli build/bin/split-file build/bin/UnicodeNameMappingGenerator build/bin/llvm-debuginfo-analyzer build/bin/llvm-yaml-parser-fuzzer build/bin/c-arcmt-test build/bin/llvm-isel-fuzzer build/bin/llvm-split build/bin/llvm-c-test build/bin/llvm-rust-demangle-fuzzer build/bin/clang-tblgen

Appreciate the help provided!
thanks,
Vrukesh

1 Like