tot clang/llvm and tot gcc performance comparision

Hi, I have looked at the LLVM code generation quality using small test cases and in general it is better than I thought and in some cases better than gcc. However, there are still some gap in SPEC performance. I have not looked at the root cause of those gaps. Anyone who cares about LLVM performance need to take this seriously.

For fair comparison, I used -fno-strict-aliasing in gcc to turn off type based aliasing. I also used -fomit-frame-pointer for clang/llvm because this is not the default. The base option is -O2. Measured on 2.4GHz core-2 box.

64bit:

llvm gcc gcc vs LLVM
164.gzip 1268 1320 4.15%
175.vpr 1605 1534 -4.42%
176.gcc 2203 2315 5.08%
181.mcf 1625 1737 6.85%
186.crafty 2411 2307 -4.30%
197.parser 1173 1166 -0.57%
252.eon 2245 2464 9.72%
253.perlbmk 2214 2444 10.37%
254.gap 1987 1978 -0.47%
255.vortex 2497 2422 -3.00%
256.bzip2 1585 1740 9.80%
300.twolf 2294 2281 -0.58%

As you can see, LLVM beats gcc in vpr, crafty and vortex by about 3%, but falls behind gcc in gzip, gcc, mcf, eon, perfbmk, and bzip2 by a large margin.

32bit:
llvm gcc gcc vs llvm

164.gzip 1210 1239 2.44%
175.vpr 1662 1621 -2.42%
181.mcf 2733 3109 13.75%
186.crafty 1812 1721 -5.00%
197.parser 1328 1289 -2.92%
253.perlbmk 2086 2580 23.67%
254.gap 1968 1912 -2.86%
255.vortex 1842 1965 6.66%
256.bzip2 1440 1553 7.82%
300.twolf 2284 2213 -3.08%

It beats gcc in vpr, crafty, parser, gap, and twolf by ~3%, but falls behind gcc in gzip, mcf, perlbmk, vortex, and bzip2. Some of them have very large performance gap. eon and gcc mis-compile.

Thanks,

David

Thanks David. Unfortunately many of us cannot use GPL v3 gcc so it’s hard for us to investigate this. One question, can you tell if gcc is inlining significantly more than llvm? We have reports that this is one of the issue plaguing eon performance.

There are also some relatively well known spec optimizations that we haven’t implemented. e.g. http://gcc.gnu.org/wiki/summit2010?action=AttachFile&do=get&target=meissner2.pdf Are there more?

Evan

Thanks David. Unfortunately many of us cannot use GPL v3 gcc so it’s hard for us to investigate this. One question, can you tell if gcc is inlining significantly more than llvm?

Evan, I have not looked at the details, so I am not sure.

We have reports that this is one of the issue plaguing eon performance.

For C++ program, inliner can be a big differentiator. For C programs, without profile feedback, the inliner probably won’t play as big a role. I looked at some small cases, llvm seems quite aggressive in inlining.

There are also some relatively well known spec optimizations that we haven’t implemented. e.g. http://gcc.gnu.org/wiki/summit2010?action=AttachFile&do=get&target=meissner2.pdf Are there more?

The paper talks about SPEC06 programs. I hope neither gcc nor llvm resorts too much on benchmark specific tunings …

Thanks,

David

Hi Evan,

Thanks David. Unfortunately many of us cannot use GPL v3 gcc so it's hard for us
to investigate this. One question, can you tell if gcc is inlining significantly
more than llvm? We have reports that this is one of the issue plaguing eon
performance.

are you allowed to look at assembler output by GPL v3 gcc? If so, maybe someone
else can do the compiling for you, and just send you results and assembler?

Ciao,

Duncan.

Hi Evan,

Thanks David. Unfortunately many of us cannot use GPL v3 gcc so it's hard for us
to investigate this. One question, can you tell if gcc is inlining significantly
more than llvm? We have reports that this is one of the issue plaguing eon
performance.

are you allowed to look at assembler output by GPL v3 gcc? If so, maybe someone
else can do the compiling for you, and just send you results and assembler?

Looking at assembly is probably ok. What you described sounds like a great improvement but it would be even better is someone can do some in-depth analysis.

Evan

Hi Evan,

Thanks David. Unfortunately many of us cannot use GPL v3 gcc so it's hard for us
to investigate this. One question, can you tell if gcc is inlining significantly
more than llvm? We have reports that this is one of the issue plaguing eon
performance.

are you allowed to look at assembler output by GPL v3 gcc? If so, maybe someone
else can do the compiling for you, and just send you results and assembler?

Looking at assembly is probably ok. What you described sounds like a great improvement but it would be even better is someone can do some in-depth analysis.

it would of course be best if someone without GPL v3 restrictions could analyse
the spec results [*]. If no-one steps forward, I could set up some kind of
build server to which you can submit preprocessed source: it would compile it
using gcc-4.5 and send you back the assembler (or something along those lines).

Ciao,

Duncan.

[*] Presumably Apple management can understand the usefulness of studying the
competition. Perhaps someone can be appointed as "GPL v3 guy", allowed to poke
at GPL v3 software to some extent.