Clang produces a much more inefficient binary (azahar) as compared to GCC

IR & Optimizations
1

Hello!

I recently decided to benchmark azahar, a 3DS emulator with both GCC and Clang to conclude the best flags and compiler for the project, I found that using O3 (without modifications) and Full LTO was the best for both compilers for this specific project.

But I happened to come across that Clang was producing severely worse code than GCC, specifically hurting the CPU caches, using 2x the cycles, executing 43% more instructions.

Do note that “Generic” binaries means compiled for such CPU, means compiled with -march=x86-64 -mtune=generic, so as to rule out ISA and architecture-specific quirkiness.

Here are the benchmark results: Clang produces a much worse binary than GCC - Pastebin.com

benchmark_script.sh

#!/bin/bash

gamemoderun azahar -f “$1” &
PID=$!
sleep 15 # Wait for game to initialize, to get more accurate data.
perf stat -d -d -p “$PID” – sleep 25

Here are the binaries themselves: [Moderator note: binaries are not provided by the LLVM project, use at your own risk] https://archive.org/details/azahar-2125.1.2-GCC-O3-generic

Extra info:

  • All binaries compiled with same flags otherwise stated in the data, though GCC used Mold and Clang used LLD (linkers shouldn’t affect runtime performance).
  • Both LLVM and GCC are 21.1.8 and GCC 16.1.1+r346, respectively.
  • Benchmarks were conducted with the emulator running idle at a scene where I got 25 FPS max, the frametimes were similar for all instances.
  • Benchmarks were conducted for a fixed 25 seconds time window.
  • Specs: Arch Linux, Core 2 Quad Q9550 @ 3.33GHz, DDR2 800 MT/s Dual Channel, GT 740 (470xx), SATA 2 SSD.

Please ask for any other information if needed! Also, I will report this to LLVM upstream after confirming this is an issue with LLVM itself.

Thanks!

2

A big performance difference on a benchmark is a valid issue, but keep your expectations low. Reducing testcases from a benchmark is a lot of work by itself, and I’m not sure anyone will take the time.

If you’re going to report an issue, please make sure to include complete build steps and benchmarking steps, including where to get all the necessary components.

2 Likes
3

If you are interested in that profiling which specific function in the program is slower (if any) is a good start

1 Like
4

It will be pretty helpful for both the community and yourself to reduce/analyze the benchmark case

1 Like
5

I will try to see what I can find today, or in the next few days, I will try to find a specific function or so as requested by gbaraldi.

Though I have noticed that the LLC miss rate is much lower than GCC’s, and L1D cache miss rate is much higher. It seems to be that LLVM is conflating a hot function too much that it fails to fit into the L1 cache, and goes straight to L2? Maybe PGO would help in this matter? I will try PGO as well, as maybe it could make the codegen less aggressive.

EDIT: I tried PGO, it didn’t help. Though the profile was pretty small, only a few minutes of data, usually with such data, it racks up a few fps, means the profile did work, the data was worth about 12MiB.

6

Also a thing to note, this is not an issue with -O3, as -O2 turned out to be actually worse for performance, for both compilers.

7

I suppose you mean L1I misses? L1D miss increases are not likely caused by overinlining. Definitely try PGO though.

8

Yep, I did try PGO, and it didn’t help, which is stated just 2 posts above.
And no, it isn’t L1I, but rather L1D, for numbers, please check the pastebin link.

My exams are very close, and I have tons of things to catch up on, and that’s why my responses are really slow, so please don’t mind if I don’t respond or don’t report results as soon enough! At least for a week or two.

9

It would be best to try newer -march and CPU, since your system is really quite outdated. I suspect default tuning model is probably not optimized for such old uarch.

10

Sorry for not stating earlier, but that doesn’t help, the PGO results were indeed compiled using -march=native, and LLVM doesn’t have any specific codegen for my CPU architecture, it defaults to SandyBridge for it.

But a micro-architecture problem shouldn’t cause such issues, the gains are usually of 1-5%, therefore the loss shouldn’t exceed that margin as well, especially in such a mature toolchain.

11

I managed to find a bit of time beside my exam preparation, and have got the perf.data files for binaries produced by both compilers, using -g -march=x86-64 -mtune=generic with Release build type, means -O3 -DNDEBUG.

[Disclaimer: The files listed below are NOT provided by the LLVM project, use at your own risk]

I assure you these are not dangerous, personally

I assure you personally, I have not injected a virus or anything like that into the files. I just added the following comment so I can keep my ability to edit the post, because I can’t edit my original post anymore, as I hadn’t added this disclaimer or so.

Here are the perf.data files and binaries: perf-data-azahar-O3-debug-generic directory listing

Although I haven’t found the time to see and compare the actual data for now, it takes quite a while for perf report to finish, and I certainly don’t have the time anymore, my exams start tomorrow.

Thanks!

12

I have extracted the output of the perf report TUI on both Clang and GCC binaries, I did so by firstly installing the respective binary for the perf-gcc.data and perf-clang.data files.

Here is the output of both clang and gcc perf.data files: GCC vs Clang perf report for Azahar - Pastebin.com

I am totally new to perf profiling and Assembly, and therefore I am using AI to speed up the research work (Exams…). Are there any specific functions catching your eye that you want me to check?

Thanks!