At the LLVM Developers' Meeting in November, I promised to work on isolating a subset of the current test suite that is useful for benchmarking. Having looked at this in more detail, most of the applications and benchmarks in the test suite are useful for benchmarking, and so I think that a better way of phrasing it is that we should construct a list of programs in the test suite that are not useful for benchmarking.

My proposed exclusion list is provided below. I constructed this exclusion list primarily based on the following experiment: I ran the test suite 10 times in three configurations: 1) On an IBM POWER7 (P7) with -O3 -mvsx, 2) On a P7 at -O0 and 3) On an Intel Xeon E5430 with -O3 all using make -j6. I then used the ministat utility (which performs a T test) to compare the timings of the two P7 configurations against each other and the Xeon configuration, requiring a detectable difference at 99.5% confidence. I looked for tests that showed no significant difference in all three comparisons. The running configuration here is purposefully noisy, the idea is to eliminate those tests that are significantly sensitive to startup time, file I/O time, memory bandwidth, etc., or just too short, and by running many tests in parallel (non-deterministically), my hope is to eliminate those tests can cannot usefully serve as benchmarks in a "normal" environment.

I'll admit being somewhat surprised by so many of the Prolangs and Shootout "benchmarks" seemingly not serving as useful benchmarks; perhaps someone can look into improving the problem size, etc. of these.

Without further ado, I propose that a test-suite configuration designed for benchmarking exclude the following:

MultiSource/Applications/Burg/burg

MultiSource/Applications/treecc/treecc

MultiSource/Benchmarks/MiBench/office-ispell/office-ispell

MultiSource/Benchmarks/MiBench/security-blowfish/security-blowfish

MultiSource/Benchmarks/Prolangs-C/allroots/allroots

MultiSource/Benchmarks/Prolangs-C/archie-client/archie

MultiSource/Benchmarks/Prolangs-C/assembler/assembler

MultiSource/Benchmarks/Prolangs-C/compiler/compiler

MultiSource/Benchmarks/Prolangs-C++/deriv1/deriv1

MultiSource/Benchmarks/Prolangs-C++/deriv2/deriv2

MultiSource/Benchmarks/Prolangs-C++/family/family

MultiSource/Benchmarks/Prolangs-C/fixoutput/fixoutput

MultiSource/Benchmarks/Prolangs-C/football/football

MultiSource/Benchmarks/Prolangs-C++/fsm/fsm

MultiSource/Benchmarks/Prolangs-C++/garage/garage

MultiSource/Benchmarks/Prolangs-C++/NP/np

MultiSource/Benchmarks/Prolangs-C++/objects/objects

MultiSource/Benchmarks/Prolangs-C++/office/office

MultiSource/Benchmarks/Prolangs-C/plot2fig/plot2fig

MultiSource/Benchmarks/Prolangs-C++/shapes/shapes

MultiSource/Benchmarks/Prolangs-C/simulator/simulator

MultiSource/Benchmarks/Prolangs-C/TimberWolfMC/timberwolfmc

MultiSource/Benchmarks/Prolangs-C++/trees/trees

MultiSource/Benchmarks/Prolangs-C/unix-smail/unix-smail

MultiSource/Benchmarks/Prolangs-C/unix-tbl/unix-tbl

MultiSource/Benchmarks/Prolangs-C++/vcirc/vcirc

SingleSource/Benchmarks/McGill/exptree

SingleSource/Benchmarks/Shootout-C++/hello

SingleSource/Benchmarks/Shootout-C++/reversefile

SingleSource/Benchmarks/Shootout-C++/spellcheck

SingleSource/Benchmarks/Shootout-C++/sumcol

SingleSource/Benchmarks/Shootout-C++/wc

SingleSource/Benchmarks/Shootout-C++/wordfreq

SingleSource/Benchmarks/Shootout/hello

SingleSource/Regression/C++/2003-05-14-array-init

SingleSource/Regression/C++/2003-05-14-expr_stmt

SingleSource/Regression/C/2003-05-14-initialize-string

SingleSource/Regression/C/2003-05-21-BitfieldHandling

SingleSource/Regression/C/2003-05-21-UnionBitfields

SingleSource/Regression/C/2003-05-21-UnionTest

SingleSource/Regression/C/2003-05-22-LocalTypeTest

SingleSource/Regression/C/2003-05-22-VarSizeArray

SingleSource/Regression/C/2003-05-23-TransparentUnion

SingleSource/Regression/C++/2003-06-08-BaseType

SingleSource/Regression/C++/2003-06-08-VirtualFunctions

SingleSource/Regression/C++/2003-06-13-Crasher

SingleSource/Regression/C/2003-06-16-InvalidInitializer

SingleSource/Regression/C/2003-06-16-VolatileError

SingleSource/Regression/C++/2003-08-20-EnumSizeProblem

SingleSource/Regression/C++/2003-09-29-NonPODsByValue

SingleSource/Regression/C/2003-10-12-GlobalVarInitializers

SingleSource/Regression/C/2004-02-03-AggregateCopy

SingleSource/Regression/C/2004-03-15-IndirectGoto

SingleSource/Regression/C/2005-05-06-LongLongSignedShift

SingleSource/Regression/C/2008-01-07-LongDouble

SingleSource/Regression/C++/2008-01-29-ParamAliasesReturn

SingleSource/Regression/C++/2011-03-28-Bitfield

SingleSource/Regression/C/badidx

SingleSource/Regression/C/bigstack

SingleSource/Regression/C++/BuiltinTypeInfo

SingleSource/Regression/C/callargs

SingleSource/Regression/C/casts

SingleSource/Regression/C/compare

SingleSource/Regression/C/ConstructorDestructorAttributes

SingleSource/Regression/C/DuffsDevice

SingleSource/Regression/C++/EH/class_hierarchy

SingleSource/Regression/C++/EH/ConditionalExpr

SingleSource/Regression/C++/EH/ctor_dtor_count

SingleSource/Regression/C++/EH/ctor_dtor_count-2

SingleSource/Regression/C++/EH/dead_try_block

SingleSource/Regression/C++/EH/exception_spec_test

SingleSource/Regression/C++/EH/function_try_block

SingleSource/Regression/C++/EH/inlined_cleanup

SingleSource/Regression/C++/EH/recursive-throw

SingleSource/Regression/C++/EH/simple_rethrow

SingleSource/Regression/C++/EH/simple_throw

SingleSource/Regression/C++/EH/throw_rethrow_test

SingleSource/Regression/C++/fixups

SingleSource/Regression/C++/global_ctor

SingleSource/Regression/C/globalrefs

SingleSource/Regression/C++/global_type

SingleSource/Regression/C++/ofstream_ctor

SingleSource/Regression/C/pointer_arithmetic

SingleSource/Regression/C++/pointer_member

SingleSource/Regression/C++/pointer_method

SingleSource/Regression/C++/pointer_method2

SingleSource/Regression/C/PR10189

SingleSource/Regression/C/PR1386

SingleSource/Regression/C/PR491

SingleSource/Regression/C/PR640

SingleSource/Regression/C++/short_circuit_dtor

SingleSource/Regression/C/sumarray

SingleSource/Regression/C/sumarraymalloc

SingleSource/Regression/C/testtrace

SingleSource/UnitTests/2002-04-17-PrintfChar

SingleSource/UnitTests/2002-05-02-ArgumentTest

SingleSource/UnitTests/2002-05-02-CastTest

SingleSource/UnitTests/2002-05-02-CastTest1

SingleSource/UnitTests/2002-05-02-CastTest2

SingleSource/UnitTests/2002-05-02-CastTest3

SingleSource/UnitTests/2002-05-02-ManyArguments

SingleSource/UnitTests/2002-05-03-NotTest

SingleSource/UnitTests/2002-05-19-DivTest

SingleSource/UnitTests/2002-08-02-CastTest

SingleSource/UnitTests/2002-08-02-CastTest2

SingleSource/UnitTests/2002-08-19-CodegenBug

SingleSource/UnitTests/2002-10-09-ArrayResolution

SingleSource/UnitTests/2002-10-12-StructureArgs

SingleSource/UnitTests/2002-10-12-StructureArgsSimple

SingleSource/UnitTests/2002-10-13-BadLoad

SingleSource/UnitTests/2002-12-13-MishaTest

SingleSource/UnitTests/2003-04-22-Switch

SingleSource/UnitTests/2003-05-02-DependentPHI

SingleSource/UnitTests/2003-05-07-VarArgs

SingleSource/UnitTests/2003-05-12-MinIntProblem

SingleSource/UnitTests/2003-05-14-AtExit

SingleSource/UnitTests/2003-05-26-Shorts

SingleSource/UnitTests/2003-05-31-CastToBool

SingleSource/UnitTests/2003-05-31-LongShifts

SingleSource/UnitTests/2003-07-06-IntOverflow

SingleSource/UnitTests/2003-07-08-BitOpsTest

SingleSource/UnitTests/2003-07-09-LoadShorts

SingleSource/UnitTests/2003-07-09-SignedArgs

SingleSource/UnitTests/2003-07-10-SignConversions

SingleSource/UnitTests/2003-08-05-CastFPToUint

SingleSource/UnitTests/2003-08-11-VaListArg

SingleSource/UnitTests/2003-08-20-FoldBug

SingleSource/UnitTests/2003-09-18-BitFieldTest

SingleSource/UnitTests/2003-10-13-SwitchTest

SingleSource/UnitTests/2003-10-29-ScalarReplBug

SingleSource/UnitTests/2004-02-02-NegativeZero

SingleSource/UnitTests/2004-06-20-StaticBitfieldInit

SingleSource/UnitTests/2004-11-28-GlobalBoolLayout

SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls

SingleSource/UnitTests/2005-05-12-Int64ToFP

SingleSource/UnitTests/2005-05-13-SDivTwo

SingleSource/UnitTests/2005-07-17-INT-To-FP

SingleSource/UnitTests/2005-11-29-LongSwitch

SingleSource/UnitTests/2006-01-29-SimpleIndirectCall

SingleSource/UnitTests/2006-02-04-DivRem

SingleSource/UnitTests/2006-12-01-float_varg

SingleSource/UnitTests/2006-12-04-DynAllocAndRestore

SingleSource/UnitTests/2006-12-07-Compare64BitConstant

SingleSource/UnitTests/2006-12-11-LoadConstants

SingleSource/UnitTests/2007-01-04-KNR-Args

SingleSource/UnitTests/2007-03-02-VaCopy

SingleSource/UnitTests/2007-04-25-weak

SingleSource/UnitTests/2008-04-18-LoopBug

SingleSource/UnitTests/2008-04-20-LoopBug2

SingleSource/UnitTests/2008-07-13-InlineSetjmp

SingleSource/UnitTests/2009-04-16-BitfieldInitialization

SingleSource/UnitTests/2009-12-07-StructReturn

SingleSource/UnitTests/2010-05-24-BitfieldTest

SingleSource/UnitTests/AtomicOps

SingleSource/UnitTests/block-byref-cxxobj-test

SingleSource/UnitTests/block-byref-test

SingleSource/UnitTests/block-call-r7674133

SingleSource/UnitTests/block-copied-in-cxxobj

SingleSource/UnitTests/block-copied-in-cxxobj-1

SingleSource/UnitTests/blockstret

SingleSource/UnitTests/byval-alignment

SingleSource/UnitTests/conditional-gnu-ext

SingleSource/UnitTests/conditional-gnu-ext-cxx

SingleSource/UnitTests/DefaultInitDynArrays

SingleSource/UnitTests/FloatPrecision

SingleSource/UnitTests/initp1

SingleSource/UnitTests/member-function-pointers

SingleSource/UnitTests/printargs

SingleSource/UnitTests/SignlessTypes/cast2

SingleSource/UnitTests/SignlessTypes/cast-bug

SingleSource/UnitTests/SignlessTypes/ccc

SingleSource/UnitTests/SignlessTypes/div

SingleSource/UnitTests/SignlessTypes/factor

SingleSource/UnitTests/SignlessTypes/Large/cast

SingleSource/UnitTests/SignlessTypes/shr

SingleSource/UnitTests/stmtexpr

SingleSource/UnitTests/StructModifyTest

SingleSource/UnitTests/TestLoop

SingleSource/UnitTests/Threads/2010-12-08-tls

SingleSource/UnitTests/Threads/tls

SingleSource/UnitTests/Vector/build

SingleSource/UnitTests/Vector/divides

SingleSource/UnitTests/Vector/sumarray

SingleSource/UnitTests/Vector/sumarray-dbl

SingleSource/UnitTests/vla

SingleSource/UnitTests/Vector/Altivec/2007-01-07-lvsl-lvsr-Regression

SingleSource/UnitTests/Vector/Altivec/alti.sdot

SingleSource/UnitTests/Vector/Altivec/casts

SingleSource/UnitTests/Vector/Altivec/test1

SingleSource/UnitTests/ms_struct-bitfield

SingleSource/UnitTests/ms_struct-bitfield-1

SingleSource/UnitTests/ms_struct-bitfield-init

SingleSource/UnitTests/ms_struct-bitfield-init-1

SingleSource/UnitTests/ms_struct_pack_layout

SingleSource/UnitTests/ms_struct_pack_layout-1

If anyone has any proposed modifications to this list, please let me know. Otherwise, I'll construct a patch to the test-suite makefiles and mail it for review soon.

Thanks in advance,

Hal