GCC 4.6, LLVM/Clang 2.9, DragonEgg Five-System Benchmarks

Just stumbled upon this and thought that someone might find it interesting
http://www.phoronix.com/scan.php?page=article&item=gcc_46_llvm29

Can anyone explain significant superiority of DragonEgg over Clang is some tests?

Some of the tests are unoptimized compiles, apparently by accident,
which I believe accounts for all the really dramatic differences. If you're
interested in knowing why the unoptimized speeds vary so much, that's
pretty simple. DragonEgg's -O0 isn't "really" -O0, because it lowers
starting from a GCC IR that's already undergone SSA transformation,
which means it's already had a number of fundamental data-flow
optimizations applied. Clang's -O0 code is noticeably worse than GCC's
because, in addition to disabling all IR and machine-level optimization,
we also use cheap-but-terrible instruction selection and register
allocation, whereas (IIRC) GCC uses its normal isel and RA algorithms.

John.

IIRC someone on #llvm mentioned that clang's lack of OpenMP support is
likely the reason for some of the most significant differences, where
dragonegg is twice as fast as clang on a duo-core, four times as fast
on a quad-core, and so on.

Yes, that's what I also believe.

Here a link to the original test results:

http://openbenchmarking.org/result/1103264-IV-1103253IV95

And here some of the test descriptions:

GraphicsMagick: This is a test of GraphicsMagick with its >>OpenMP<< implementation that performs various imaging tests to stress the system's CPU.

Cheers
Tobi

Himeno is built -O0, which makes zero sense as a performance benchmark. And you are right GraphicsMagick uses OpenMP, which means the comparison between Clang and the rest of the compilers are totally meaningless. 7-Zip builds just fine for me with Clang, I have no idea why they couldn't get it to build.

I think it's worthwhile to relay these information and hopefully they will correct these mistakes. But color me skeptical.

Evan

I sent him email and received not much in the way of a hopeful response.

-eric

Eric Christopher wrote:

Eric Christopher wrote:

Himeno is built -O0, which makes zero sense as a performance benchmark. And you are right GraphicsMagick uses OpenMP, which means the comparison between Clang and the rest of the compilers are totally meaningless. 7-Zip builds just fine for me with Clang, I have no idea why they couldn't get it to build.

I think it's worthwhile to relay these information and hopefully they will correct these mistakes. But color me skeptical.
   
I sent him email and received not much in the way of a hopeful response.

phoronix is an ad machine and solely strives to make the graphs more interesting.... I've added Michael on cc and who has always corrected blatant errata for me in the past.

I've talked with Michael over email. He was very polite, I just don't know if anything was going to be mentioned or changed.

With regard to OpenMP - Did it say if the -omp flags were used? If not then I don't think gcc automatically turns it on.

It should be on for an openmp enabled GraphicsMagick by default if it detects you are gcc. I should think at least.

-eric

Eric Christopher wrote:

Himeno is built -O0, which makes zero sense as a performance benchmark. And you are right GraphicsMagick uses OpenMP, which means the comparison between Clang and the rest of the compilers are totally meaningless. 7-Zip builds just fine for me with Clang, I have no idea why they couldn't get it to build.

Yes, I've already discussed some of this with Eric in a private thread the other day.

While the latest upstream Himeno release will be pulled, which evidently is now multi-threaded and uses -O3, I would argue though that in the context of benchmarking defaults, even '-O0' does matter if looking at the compiler performance. Such as for software projects that may not specify any optimization flag, I really doubt __most users__ when pulling down new code are checking what a given program is using as their compiler flags.

I think it's worthwhile to relay these information and hopefully they will correct these mistakes. But color me skeptical.

I sent him email and received not much in the way of a hopeful response.

phoronix is an ad machine and solely strives to make the graphs more interesting.... I've added Michael on cc and who has always corrected blatant errata for me in the past.

I've talked with Michael over email. He was very polite, I just don't know if anything was going to be mentioned or changed.

Yes, the Himeno test will be updated against the latest upstream release as soon as time allows, etc.

With regard to OpenMP - Did it say if the -omp flags were used? If not then I don't think gcc automatically turns it on.

It should be on for an openmp enabled GraphicsMagick by default if it detects you are gcc. I should think at least.

-eric

Right, GraphicsMagick uses OpenMP and as discussed in the private exchange with Eric the other days, that's likely why Clang is behind GCC.

Hi Michael,

I personally don't agree that -O0 matters in this context, but it doesn't really matter what I think :).

On request though: the web site just presents numbers, and leaves it up to the reader to draw their own conclusion, as though all of the numbers were measuring the same thing and reflected on the compilers in an arbitrary way. I think it would be a great service to your users to say things like "and we see here that clang doesn't support OpenMP, which is why it has terrible multicore performance on that benchmark" for example.

Does it sound reasonable to mention things like OpenMP and -O0 performance?

-Chris

Hi Chris,

Yes, it's certainly reasonable to include such information when it's brought up. I had added the OpenMP GraphicsMagick note a short while ago, but I've now added the Himeno note as well to avoid confusion.

Thanks for the feedback, as always.

-- Michael

Default CFLAGS for most open source projects are "-O2 -g". Building with
"-O0" is unreasonable unless it is a development build.