Hi All !
Recently in the mailing list there was the question about benchmarking
LLVM. I was told that LLVM get's benchmarked in the nightly test.
While this is true, I wanted to have a tool to compare LLVM against
other compilers, so I wrote a little python program (attached) that
filled my need.
It is completely outside of the LLVM makefile framework, but this stems
from the fact that I can use it to compare any GCC-like compiler, e.g.
gcc-4.2 against Intel ICC. It also allows one to compare the generated
code sizes, which can be quite nice.
Here is a sample output of a boring run on hello programs:
$ ./ccbench -v
Checking compilers ...
Compiler /usr/bin/g++-3.3 not found
g++-4.1: g++-4.1 (GCC) 4.1.3 20070812 (prerelease) (Debian 4.1.2-15)
g++-4.2: g++-4.2 (GCC) 4.2.1 (Debian 4.2.1-3)
llvm-g++: llvm-g++ (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build )
llvm-g++-bc: llvm-g++ (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build )
icc: icc (ICC) 10.0 20070809
gcc-3.3: gcc-3.3 (GCC) 3.3.6 (Debian 1:3.3.6-15)
gcc-4.1: gcc-4.1 (GCC) 4.1.3 20070812 (prerelease) (Debian 4.1.2-15)
gcc-4.2: gcc-4.2 (GCC) 4.2.1 (Debian 4.2.1-3)
llvm-gcc: llvm-gcc (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build )
llvm-gcc-bc: llvm-gcc (GCC) 4.0.1 (Apple Computer, Inc. build 5449)(LLVM build )
Benchmark runtime comp code data bss compiler options
hello.cpp 0.01 1.20 1904 320 148 g++-4.1 -O0
hello.cpp 0.01 1.24 2365 328 148 g++-4.2 -O0
hello.cpp 0.01 1.36 353236 11103 23736 llvm-g++ -O0
hello.cpp 0.01 1.23 1499 300 148 llvm-g++-bc -O0
hello.cpp 0.01 1.27 2186 320 148 icc -O0
hello.c 0.01 0.29 892 284 4 icc -O0
hello.c 0.01 0.20 852 260 4 gcc-3.3 -O0
hello.c 0.01 0.24 844 268 4 gcc-4.1 -O0
hello.c 0.01 0.24 848 268 4 gcc-4.2 -O0
hello.c 0.01 0.20 852 256 4 llvm-gcc -O0
hello.c 0.01 0.25 860 268 4 llvm-gcc-bc -O0
hello.cpp 0.01 1.17 1765 316 148 g++-4.1 -O2
hello.cpp 0.01 1.20 1837 316 148 g++-4.2 -O2
hello.cpp 0.01 1.44 353204 11103 23736 llvm-g++ -O2
hello.cpp 0.01 1.36 1380 296 148 llvm-g++-bc -O2
hello.cpp 0.01 1.25 10210 996 1252 icc -O2
hello.c 0.01 0.29 8949 960 1092 icc -O2
hello.c 0.01 0.21 852 260 4 gcc-3.3 -O2
hello.c 0.01 0.23 860 268 4 gcc-4.1 -O2
hello.c 0.01 0.23 864 268 4 gcc-4.2 -O2
hello.c 0.01 0.20 836 256 4 llvm-gcc -O2
hello.c 0.01 0.25 860 268 4 llvm-gcc-bc -O2
Total benchmark time 24.31s
Description:
Benchmark: basename of the benchmark that has been compiled
runtime: minimum time of execution for the benchmark progra
comp: minimum compile time
code: code size (.text segment of resulting ELF executable)
data: data size (.data segment of resulting ELF executable)
bss: zero-initialized data size (.bss segment ...)
compiler: short-name for the compiler. llvm-g??-bc means that
the program was compiled with --emit-llvm and linked
with llvm-ld
options: optimization options used for this run
You can specify the optimization options to be used on the command
line, e.g.
$ ./ccbench -- "-O2" "-O2 -march=pentium4"
Other command line options are available, see --help.
The list of benchmarks to be compiled and run has to be specified inside
the source code, look at the end of the program.
NOTE: the "llvm-g++" produces VERY large programs for any program that
uses libstdc++, because it was compiled with --disable-shared.
Here is an example where you can see the effect of the "-Os" compiler switch
to various C++ programs:
$ ./ccbench -- "-O0" "-O2" "-Os" | sort
bigfib.cpp 0.16 5.28 18763 432 572 g++-4.2 -O2
bigfib.cpp 0.17 2.42 371805 12589 23736 llvm-g++ -Os
bigfib.cpp 0.17 3.75 26810 1116 1380 icc -Os
bigfib.cpp 0.17 3.85 28902 1116 1380 icc -O2
bigfib.cpp 0.17 4.48 18902 444 572 g++-4.1 -O2
bigfib.cpp 0.18 2.30 7403 376 156 llvm-g++-bc -Os
bigfib.cpp 0.18 2.74 11838 380 420 llvm-g++-bc -O2
bigfib.cpp 0.18 2.82 374391 12589 23736 llvm-g++ -O2
bigfib.cpp 0.22 2.21 8090 388 156 llvm-g++-bc -O0
bigfib.cpp 0.29 2.99 14101 424 312 g++-4.1 -Os
bigfib.cpp 0.29 3.00 12419 424 312 g++-4.2 -Os
bigfib.cpp 0.59 1.96 20897 448 312 g++-4.2 -O0
bigfib.cpp 0.71 2.05 24029 440 312 g++-4.1 -O0
bigfib.cpp 0.72 2.05 24029 440 312 g++-4.1 -O0
bigfib.cpp 0.87 2.71 385897 12589 23736 llvm-g++ -O0
bigfib.cpp 0.91 2.03 42886 440 312 icc -O0
mandel-text.cpp 0.06 0.38 1381 296 4 g++-4.1 -Os
mandel-text.cpp 0.06 0.39 1385 296 4 g++-4.2 -Os
mandel-text.cpp 0.06 0.39 1483 296 8 g++-4.1 -O2
mandel-text.cpp 0.06 0.41 1491 296 8 g++-4.2 -O2
mandel-text.cpp 0.06 0.46 9505 972 1092 icc -Os
mandel-text.cpp 0.06 0.51 9621 972 1092 icc -O2
mandel-text.cpp 0.11 0.26 1346 264 4 llvm-g++ -O0
mandel-text.cpp 0.11 0.36 1533 296 4 g++-4.1 -O0
mandel-text.cpp 0.12 0.35 1533 296 4 g++-4.1 -O0
mandel-text.cpp 0.12 0.36 1537 296 4 g++-4.2 -O0
mandel-text.cpp 0.14 0.47 1219 276 4 llvm-g++-bc -O0
mandel-text.cpp 0.14 0.48 1309 276 8 llvm-g++-bc -O2
mandel-text.cpp 0.14 0.49 1309 276 8 llvm-g++-bc -Os
mandel-text.cpp 0.23 0.26 1260 264 8 llvm-g++ -O2
mandel-text.cpp 0.24 0.26 1260 264 8 llvm-g++ -Os
mandel-text.cpp 0.27 0.47 1636 296 4 icc -O0
oopack_v1p8.cpp 0.62 1.07 14580 1288 118500 icc -O2
oopack_v1p8.cpp 0.66 0.88 5398 352 116256 g++-4.1 -O2
oopack_v1p8.cpp 0.66 0.94 5630 352 116256 g++-4.2 -O2
oopack_v1p8.cpp 0.76 0.97 13909 1292 118500 icc -Os
oopack_v1p8.cpp 2.05 0.61 51011 1207 132940 llvm-g++ -Os
oopack_v1p8.cpp 2.07 0.60 50874 1207 132940 llvm-g++ -O2
oopack_v1p8.cpp 2.28 0.85 5320 356 116256 g++-4.1 -Os
oopack_v1p8.cpp 2.28 0.93 5352 356 116176 g++-4.2 -Os
oopack_v1p8.cpp 2.54 0.82 4636 340 116196 llvm-g++-bc -O2
oopack_v1p8.cpp 2.63 0.84 4668 340 116196 llvm-g++-bc -Os
oopack_v1p8.cpp 2.65 0.75 5337 312 116276 llvm-g++-bc -O0
oopack_v1p8.cpp 4.53 0.59 53582 1207 132972 llvm-g++ -O0
oopack_v1p8.cpp 4.92 0.59 7195 352 116256 g++-4.2 -O0
oopack_v1p8.cpp 4.97 0.58 7215 352 116256 g++-4.1 -O0
oopack_v1p8.cpp 5.45 0.70 8856 640 117440 icc -O0
stepanov_v1p2.cpp 1.01 1.09 64931 1600 33408 icc -O2
stepanov_v1p2.cpp 1.02 1.00 64471 1600 33408 icc -Os
stepanov_v1p2.cpp 1.02 1.17 6519 340 32352 g++-4.2 -O2
stepanov_v1p2.cpp 1.07 1.18 6555 340 32352 g++-4.1 -O2
stepanov_v1p2.cpp 3.06 0.68 6082 308 32420 llvm-g++ -Os
stepanov_v1p2.cpp 3.38 0.69 5839 308 32420 llvm-g++ -O2
stepanov_v1p2.cpp 3.88 0.80 4391 308 32384 llvm-g++-bc -O0
stepanov_v1p2.cpp 4.86 1.06 5726 340 32280 g++-4.2 -Os
stepanov_v1p2.cpp 5.37 0.97 5770 340 32352 g++-4.1 -Os
stepanov_v1p2.cpp 5.38 0.87 3972 304 32192 llvm-g++-bc -O2
stepanov_v1p2.cpp 5.88 0.88 4004 304 32192 llvm-g++-bc -Os
stepanov_v1p2.cpp 25.60 0.73 8802 340 32352 g++-4.2 -O0
stepanov_v1p2.cpp 26.56 0.72 8918 340 32352 g++-4.1 -O0
stepanov_v1p2.cpp 30.56 0.69 10473 308 32420 llvm-g++ -O0
stepanov_v1p2.cpp 34.39 0.86 74167 1600 33408 icc -O0
Because you might want to sort such output according to your own criteria,
you could call ./ccbench with --csv and load the resulting CSV (comma
separated values) into your favourite spreadsheet or SQL database.
ccbench (16.2 KB)