llvm-cov accepting many binary files for aggregated coverage reports

Hi All,

I want to provide a solution that presents code coverage reports that include the aggregated code-counts across many binaries. Our test engineers currently do this using gcov-mode by merged .gcda data files. We can do a similar merge of the .profraw files, so that the many binaries are represented in one .profdata file; However, llvm-cov will only generate reports based on one binary at a time.

My suggested solution to this problem is for llvm-cov to accept multiple binary files on the command line. Thereby allowing every coverage-count / source file (that is compiled into at least one binary) to be represented in the report from a single call to llvm-cov. Presently the command format is: llvm-cov show [options] <executable | object file> . I suggest adding the option “-bin=<executable | object file>” so that more than one binary (or object file) can be listed as: -bin=binary1.elf -bin=binary2.elf -bin=binary3.elf.

I would like people’s opinions first so that may be acceptable when I come to upstream this.

Thanks

If many binaries share some library code, it can be confusing to show the aggregated coverage data of the library as if they are valid for any individual binary.

It seems to me that a per-library coverage report is more appropriate here. llvm can take DSO, but not yet static archive file (.a) – so I can see it being a useful feature if you can teach llvm-cov to handle it. On the other hand, given a set of standalone object files, I don’t see why a wrapper script (that passes object file to llvm-cov one by one) won’t work well – so I am not convinced that the new option proposed is in general useful.

thanks,

David

Hi David,

In my use case, our test teams want a single html index file that links to an aggregated coverage report that covers an entire code repository. I see that there is a new patch of generating html reports, is there a way to index all of the individual source files into one summary report, where there are many binary files to consider?

> I don’t see why a wrapper script (that passes object file to llvm-cov one by one) won’t work well

If the functionality to generate the toplevel html index page for one binary is embedded in llvm-cov, how will a wrapper script generate the toplevel html index page for a code-repository that is tested across multiple binaries.

> Ifmany binaries share some library code, it can be confusing to show the aggregated coverage data of the library as if they are valid for any individual binary.

The functionality to do this merge of profile data across many binaries already exists in llvm-profdata. Our test engineers do not find this confusing, on the contrary they find it very helpful, where they are looking for an aggregated coverage report.

> It seems to me that a per-library coverage report is more appropriate here.

I’m not sure how this differs from generating a single report across an entire code-repository; passing the code-repository as a single library is just one use-case. There are also use-cases where a single repository is compiled into many libraries; do you plan to support giving llvm-cov multiple library files (or multiple object files)?

Ultimately, in my case, I would like to see a single toplevel coverage report (preferably in html) and all of the detailed reports for each source file that includes the coverage from many binaries.

Thanks,
Phillip

Ok – this sounds like a reasonable use case so personally I don’t have strong objection to adding this feature (assuming it does not add too much additional complexity to llvm-cov implementation). Perhaps we can discuss it more in the actual code review?

thanks,

David

Thanks David, I will look at the additional complexity for adding this feature.

Maggie