LLVM versus Intel's PIN tool

Hello,

I am trying to decide between using LLVM and Intel's PIN
tool as the dynamic optimization tool for my PhD thesis.
Specifically, the tool I choose has to have the following characteristics:

1. I should be able to run all the Spec 2000 and Spec 95 floating point
and integer benchmarks.

2. I should be able to instrument the beginning and end of specific program regions (e.g., functions, loops) with calls to subroutines that I write.

3. I need to be able to patch the binary code of an executing program. Specifically, I want to be able to insert NOPs (into functions and loops) at compile time that I patch into calls to subroutines at runtime.

My questions are:

A. The testresults at http://llvm.cs.uiuc.edu/testresults/X86/ show that several benchmarks (swim, applu, tomcatv, su2cor...) in the Spec CFP 2000 and 95 suites failed when run under the JIT. Why did they fail and what are the limitations in running Spec benchmarks?

B. Is what I am suggesting in 3 possible in LLVM?

C. Has anyone here used both LLVM and Intel's PIN? If so, can you comment on the drawbacks of using one over the other? I am especially interested in any comparisons of their performance and ease-of-use.

Thanks,

Vasanth

Hi Vasanth,

Can you be a little more precise about what you mean by "dynamic
optimization"? In one sense, neither LLVM nor Pin are dynamic optimization
tools per se - LLVM is more of a compiler(-building) suite while Pin is an
instrumentation toolkit.

1. I should be able to run all the Spec 2000 and Spec 95 floating point
and integer benchmarks.

LLVM will do this - if it doesn't (on your particular platform) then
that's a bug that can and will be fixed. :slight_smile: Pin just runs whatever
binaries you have managed to build (perhaps even using LLVM.)

2. I should be able to instrument the beginning and end of specific
program regions (e.g., functions, loops) with calls to subroutines that I
write.

OK. You could do this with the LLVM JIT - easily if you want to patch the
heads and tails of functions, a little more work (but not too much) to get
at specific loops. With Pin you won't have any issues, but either way
you'll be writing a bit of code.

3. I need to be able to patch the binary code of an executing program.
Specifically, I want to be able to insert NOPs (into functions and
loops) at compile time that I patch into calls to subroutines at runtime.

Do you really need to insert NOPs and then write over them? Can't you just
replace calls to functions with calls to new functions that in turn call
whatever subroutines you like?

My questions are:

A. The testresults at http://llvm.cs.uiuc.edu/testresults/X86/ show that
several benchmarks (swim, applu, tomcatv, su2cor...) in the Spec CFP 2000
and 95 suites failed when run under the JIT. Why did they fail and what
are the limitations in running Spec benchmarks?

I'm not sure why these don't run in the JIT, actually. They should.
(Anyone?)

B. Is what I am suggesting in 3 possible in LLVM?

You should be able to achieve the same effect, but I'm not sure if it will
actually involve overwriting NOPs (if for no other reason than an
optimizing compiler like LLVM doesn't emit NOPs lightly! :wink:

C. Has anyone here used both LLVM and Intel's PIN? If so, can you
comment on the drawbacks of using one over the other? I am especially
interested in any comparisons of their performance and ease-of-use.

I have used LLVM and the old (Itanium-only) version of Pin. It's hard to
compare these directly as they're quite different things. One can't really
compare their performance as they do different things: LLVM is an
optimizing compiler, Pin is an instrumentation tool. As for ease-of-use,
I'd say they're both reasonably easy to use, but it really depends on what
you want to do. If by "dynamic optimization" you are actually thinking of
*optimizing* (higher-level) code, LLVM is probably the way to go. If you
are more interested in *binary* reoptimization, however, you could
certainly use Pin to help you find and characterize interesting hot spots
but beyond that you'll be on your own.

       Hope this helps,

       Duraid

P.S. If you are feeling insane, you can probably figure out some fun way
to use both LLVM *and* Pin, e.g. you could turn LLVM into a Pin tool with
callbacks that trigger aggressive LLVM optimizations/register
allocators/whatever on hot functions.

They fail because they are not loading fortran runtime library, thus they fail with unresolved symbols. One way to deal with this would be to compile the fortran runtime library with LLVM itself (which we've done),
to teach the JIT to load .a files, or to use a different Fortran front-end.

Note that the X86 tester is using F2C to run fortran programs. It is not compatible with all of the SPEC tests (e.g. those that use FORTRAN 95). To support these, using something like the NAG fortran front-end would be a better idea (and it does work with LLVM).

-Chris