Comparison of optimizations

Hi,

I want to measure the performance of functions within a program, in
order to see whether or not a given set of optimizations are useful
(making the code faster in my case). It would be ideal to compare the
LLVM IR representation of the functions before and after the
optimizations, but the process has to be automatized so if there is no
code that can do this now then I guess it will be too hard to come up
with good heuristics for comparing the code with respect to execution
performance.
My current approach would be to use a profiler, such as
google-perftools, but the information acquired suffers from some
inaccuracy.
However I would like to hear your take on it and maybe if I should
implement the measurement mechanism in an interpreter (lli) that uses
a high resolution timer for entering and exiting functions. But it
depends on if the performance of the executed LLVM IR code in lli is
reflective on the performance from native compiled code, such that it
can be tested if a set of optimizations are useful or not. Or would it
be more desirable to make a pass that inserts code to activate and
deactivate a timer on function entry and exit?

Cheers, Michael

Maybe you could use the available profilers (i.e edge profiler, path profiler…) to generate instrumented code, enabling and disabling the optimizations you want to test, and then make the comparison (assuming you can actually disable the optimizations of interest without compromising later passes). You could modify the profilers to collect information that is more interesting for you as well.

bejer <bejerdk@gmail.com> writes:

Hi,

I want to measure the performance of functions within a program, in
order to see whether or not a given set of optimizations are useful
(making the code faster in my case). It would be ideal to compare the
LLVM IR representation of the functions before and after the
optimizations, but the process has to be automatized so if there is no
code that can do this now then I guess it will be too hard to come up
with good heuristics for comparing the code with respect to execution
performance.

Camparing IR is going to be nearly useless. It won't tell you much
about actual performance.

My current approach would be to use a profiler, such as
google-perftools, but the information acquired suffers from some
inaccuracy.

What are you trying to measure that would require a profile? The best
way to measure performance improvements of optimization is to compile
one version of the code without optimization and another with
optimization. Run both and see which is faster. It is the definitive
answer.

However I would like to hear your take on it and maybe if I should
implement the measurement mechanism in an interpreter (lli) that uses
a high resolution timer for entering and exiting functions.

But you would be measuring interpreter performance, not performance of
the code. Unless you will always run using the interpreter, this is
going to be misleading, at best.

But it depends on if the performance of the executed LLVM IR code in
lli is reflective on the performance from native compiled code, such
that it can be tested if a set of optimizations are useful or not.

It is not.

Or would it be more desirable to make a pass that inserts code to
activate and deactivate a timer on function entry and exit?

Do you just want to time one function out of some benchmark? No need to
make a pass. Just add timers to the code. But make sure that function
is really the performance dominator!

                              -Dave