Hello llvm-dev,
We recently collected some data on the test-suite using cfl=steens AA compared to BasicAA and wanted to report the results. There is one failing test (SingleSource/Benchmarks/Shootout/Shootout-lists.exec) which we are looking into. All unlisted tests show no statistical difference. There are additional details about the testing methodology at the bottom of the email.
- Brian Homerding
The format of the results is: (negative means BasicAA is better)
Test
Absolute Difference
Percentage Difference
**************** Execution Differences ****************
SingleSource/Benchmarks/Shootout/Shootout-matrix.exe
0.104083 +/- 0.0617407
7.14886% +/- 4.2406%
SingleSource/Benchmarks/CoyoteBench/huffbench.exec
0.408713 +/- 0.174997
3.21233% +/- 1.37541%
**************** Compilation Differences ****************
MultiSource/Benchmarks/DOE-ProxyApps-C/SimpleMOC/SimpleMOC.compile
-0.176327 +/- 0.0750171
-6.18574% +/- 2.63169%
MultiSource/Benchmarks/tramp3d-v4/tramp3d-v4.compile
-0.599067 +/- 0.49799
-1.23398% +/- 1.02577%
MultiSource/Benchmarks/nbench/nbench.compile
0.31194 +/- 0.137726
7.0723% +/- 3.12252%
MultiSource/Benchmarks/MallocBench/gs/gs.compile
-0.0901467 +/- 0.0737197
-0.85973% +/- 0.703066%
MultiSource/Benchmarks/MiBench/consumer-typeset/consumer-typeset.compile
-0.542867 +/- 0.511135
-1.72053% +/- 1.61996%
**************** Link Differences ****************
MultiSource/Benchmarks/TSVC/ControlFlow-dbl/ControlFlow-dbl-link.compile
-0.00284333 +/- 0.0026716
-9.62319% +/- 9.04198%
**SingleSource/Benchmarks/Shootout-C++/**Shootout-C+±ary3-link.compile
-0.0039 +/- 0.00380256
-8.40819% +/- 8.19811%
MultiSource/Benchmarks/SciMark2-C/s****cimark2-link.compile
-0.00320667 +/- 0.00242367
-12.8335% +/- 9.69987%
MultiSource/Benchmarks/tramp3d-v4/t****ramp3d-v4-link.compile
-0.00392 +/- 0.0036359
-4.30517% +/- 3.99315%
**MultiSource/Benchmarks/TSVC/NodeSplitting-dbl/**NodeSplitting-dbl-link.compile
-0.00362667 +/- 0.00287988
-12.9957% +/- 10.3197%
** Notes on the testing methodology **
We collected 30 compilation and execution samples on an Intel(R) Xeon(R) Platinum 8180M CPU @ 2.50GHz Skylake machine using revision 348151. Beyond the “-use-cfl-aa=steens” flag we also used “-march=native -O3”. The results were compared using a python version of the ministat tool.