Using lnt to run SPEC tests: possible or not?

It appears that lnt has the capability to run the SPEC2000/2006/2017(maybe) tests from looking at the options and from poking around in the mailing list archives. Or does it? I experimented a bit and cannot get it to work right. However, it isn't really documented anywhere so maybe I didn't set it up properly. Plus there's all the mention of llvmgcc...

So is anyone doing this? Is there some setup information that I missed? Thanks for any tips on how to do this!

Hi Bill,

Yes, it is possible to run SPEC tests through LNT/test-suite.
You won’t get official reportable SPEC numbers as you don’t use the official build-and-run framework from SPEC, but OTOH you get the extra features in LNT/test-suite working out of the box, e.g. the ones documented at http://blog.llvm.org/2016/06/using-lnt-to-track-performance.html.

To make use of this, I think you need to use the cmake driver in the test-suite.
Which, if you use the LNT wrapper to drive runs, is invoked using “lnt runtest test-suite”.

A few example of the important options on the command line that I hope will get you started:

  • Running SPEC2K6int reference inputs: lnt runtest test-suite --only-test=External/SPEC/CINT2006 --cmake-define TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define TEST_SUITE_RUN_TYPE=ref

  • Running SPEC2K6int train inputs: lnt runtest test-suite --only-test=External/SPEC/CINT2006 --cmake-define TEST_SUITE_SPEC2006_ROOT=/work/Benchmark/spec2006 --cmake-define TEST_SUITE_RUN_TYPE=train

  • Running SPEC2Kfp reference inputs: lnt runtest test-suite --only-test=External/SPEC/CFP2000 --cmake-define TEST_SUITE_SPEC2000_ROOT=/work/Benchmark/spec2000 --cmake-define TEST_SUITE_RUN_TYPE=ref

Thanks,

Kristof

Thanks Kristof.

I am hitting some odd cmake issues (it isn't handing pthread stuff properly for some reason) that I haven't seen before but this should be me going. I've used lnt for running the llvm test suite many times using "runtest nt" without problem.

Hmmm. powerpc64 isn't listed in the known architectures in test_suite.py so I may need to update some of these files.

If you don’t want to submit the results to an LNT server you can also just run the test-suite directly without going through LNT.

You should be able to do something like:
cmake -DCMAKE_C_COMPILER=/path/to/my/clang -DTEST_SUITE_SPECXXX_ROOT -DTEST_SUITE_SUBDIRS=External/SPECXXX -GNinja /path/to/test-suite

NOte: use ccmake or cmake -L to see a list of variables you can tweak.

ninja
llvm-lit

to build and run SPEC via the test-suite infrastructure.

  • Matthias