2018-05-25 13:49 GMT-05:00 Pankaj Kukreja via llvm-dev
I am adding some benchmarks to the test-suite as a part of my GSoC project.
I am planning to use the google benchmark library on some benchmarks. I
would like to know your opinion/suggestion on how I should proceed with this
library and how the design should be(like limiting the number of times a
kernel is executed so that overall runtime of test-suite can be controlled,
multiple inputs sizes, adding it to small kernels etc.).
I would love to hear any other suggestions that you may have.
I would like to add some details of the intent.
We would like to benchmark common algorithms that appear in many
sources (Think of linear algebra, image processing, etc.). Usually,
they only consist of a single function. Only the kernel itself is of
interest, but not e.g. the data initialization. Depending on the
optimization (e.g. by Polly, parallelization, offloading), the
execution time can vary widely.
Pankaj already added a review at
where the idea to use Google Benchmark came up. However, as such it
does not fulfill all the requirements, in particular, it does not
check for correctness.
Here is a list of things I would like to see:
- Check correct output
- Execution time
- Compilation time
- LLVM pass statistics
- Code size
- Hardware performance counters
- Multiple problem sizes
- Measuring the above for the kernel only.
- Optional very large problem sizes (that test every cache level),
disabled by default
- Repeated execution to average-out noise
- Taking cold/warm cache into account
Here is an idea on how this could be implemented:
Every benchmark consists of two files: The source for the kernel and a
driver that initializes the input data, knows how to call the kernel
in the other file and can check the correct output.
The framework recognizes all benchmarks and their drivers and does the
- Compile the driver
- Time the compilation of the kernel using -save-stats=obj
- Get the kernel's code size using llvm-size
- Link driver, kernel and Google Benchmark together.
I think you might run into artificial overhead here if you’re not careful. In particular you might run into:
- Missed in-lining opportunity in the benchmark. If you expect the kernels to be potentially inlined, this might be a problem.
- The link order might cause interference depending on the linker being used.
- If you’re doing LTO then that would add an additional wrinkle.
They’re not show-stoppers, but these are some of the things to look out for and consider.
- Instruct the driver to run the kernel with a small problem size and
check the correctness.
In practice, what I’ve seen is mixing unit tests which perform correctness checks (using Google Test/Mock) and then co-locating the benchmarks in the same file. This way you can choose to run just the tests or the benchmarks in the same compilation mode. I’m not sure whether there’s already a copy of the Google Test/Mock libraries in the test-suite, but I’d think those shouldn’t be too hard (nor controversial) to add.
- Instructs Google Benchmark to run the kernel to get a reliable
average execution time of the kernel (without the input data
There’s ways to write the benchmarks so that you only measure a small part of the actual benchmark. The manuals will be really helpful in pointing out how to do that.
In particular, you can pause the timing when you’re doing the data initialisation and then resume just before you run the kernel.
- LNT's --exec-multisample does not need to run the benchmarks
multiple times, as Google Benchmark already did so.
I thought recent patches already does some of this. Hal would know.
PS. I’d be happy to do some reviews of uses of the Google Benchmark library, if you need additional reviewers.