Hi all,
I would like to get some feedback on who in the community is using the
LLVM test-suite repository, and in what ways.
I am talking specifically about the "test-suite" SVN project, e.g.:
http://llvm.org/svn/llvm-project/test-suite/trunk/
not about the "test (no-dash) suite" that is part of LLVM.
If you are using the LLVM test-suite project directly (as opposed to
via LNT), I'd appreciate it if you could reply to me personally with
the following information:
1. What facilities of the project do you use:
[ X] User defined test functionality
[ X] Ability to execute two compilers (baseline, modified)
Ability to use Makefile's to modify and rerun small parts of tests
[ X] Test suite ability to work with LLVM bit code files and tools (as
opposed to native object files)
[ X] Ability to test SPEC
Ability to execute tests on a remote target
[ X] Ability to test Fortran codes
The ability to run Fortran code is something we've used in the past, and we may want to use it in the future (at least if there's Fortran code in SPEC 2006). However, I think it's a very low priority for us.
2. What compilers do you use the project to test:
[ X] clang
llvm-gcc
dragonegg
[ X] Other LLVM compilers (?)
By other compilers, I mean research prototypes. Some are modified versions of Clang; others are passes that are loaded into opt. Others might use llvm-gcc (although I don't know of any off-hand that do that we care about).
Other non-LLVM compilers
3. What TEST variants do you use:
I do not know what this means
nightly
simple
[ X] Other, please specify: _________
We use hand-written variants in SAFECode, Poolalloc, and in internal research projects. These Makefiles are out-of-tree but plug into the test suite using an extension mechanism that Chris Lattner originally wrote for the DSA/Poolalloc project.
We have used test-suite heavily in our research experiments and open-source project testing. The two features that are most beneficial to us are:
1) We can plug into the LLVM test suite to do customized experiments without modifying the core test-suite Makefiles.
2) The API doesn't change, so we continue using the same infrastructure with each new LLVM release. The SAFECode and Poolalloc tests were written years ago but have continued working with only minor modification. Many of our internal projects have just copied and modified what they've done. Experimental infrastructure is a pain to set up, so having most of the work done is a real benefit.
-- John T.