Poll: LLVM test-suite project usage

Hi all,

I would like to get some feedback on who in the community is using the
LLVM test-suite repository, and in what ways.

I am talking specifically about the "test-suite" SVN project, e.g.:
  http://llvm.org/svn/llvm-project/test-suite/trunk/
not about the "test (no-dash) suite" that is part of LLVM.

If you are using the LLVM test-suite project directly (as opposed to
via LNT), I'd appreciate it if you could reply to me personally with
the following information:

1. What facilities of the project do you use:
[ ] User defined test functionality
[ ] Ability to execute two compilers (baseline, modified)
[ ] Ability to use Makefile's to modify and rerun small parts of tests
[ ] Test suite ability to work with LLVM bit code files and tools (as
opposed to native object files)
[ ] Ability to test SPEC
[ ] Ability to execute tests on a remote target
[ ] Ability to test Fortran codes

2. What compilers do you use the project to test:
[ ] clang
[ ] llvm-gcc
[ ] dragonegg
[ ] Other LLVM compilers (?)
[ ] Other non-LLVM compilers

3. What TEST variants do you use:
[ ] I do not know what this means
[ ] nightly
[ ] simple
[ ] Other, please specify: _________

- Daniel

Yes, I use the test-suite (TEST.simple) as a check-in criteria on anything non trivial.

Hi all,

I would like to get some feedback on who in the community is using the
LLVM test-suite repository, and in what ways.

I am talking specifically about the "test-suite" SVN project, e.g.:
http://llvm.org/svn/llvm-project/test-suite/trunk/
not about the "test (no-dash) suite" that is part of LLVM.

If you are using the LLVM test-suite project directly (as opposed to
via LNT), I'd appreciate it if you could reply to me personally with
the following information:

1. What facilities of the project do you use:
[ ] User defined test functionality
[ ] Ability to execute two compilers (baseline, modified)
[X] Ability to use Makefile's to modify and rerun small parts of tests

  I run 'make' in sub directories to parts of the build.

[ ] Test suite ability to work with LLVM bit code files and tools (as
opposed to native object files)
[X ] Ability to test SPEC

Yes.

[ ] Ability to execute tests on a remote target
[ ] Ability to test Fortran codes

2. What compilers do you use the project to test:
[ X] clang
[ ] llvm-gcc
[ ] dragonegg
[ ] Other LLVM compilers (?)
[ ] Other non-LLVM compilers

Just Clang.

3. What TEST variants do you use:
[ ] I do not know what this means
[ ] nightly
[x ] simple
[ ] Other, please specify: _________

Just Simple.

Hi all,

I would like to get some feedback on who in the community is using the
LLVM test-suite repository, and in what ways.

I am talking specifically about the "test-suite" SVN project, e.g.:
   http://llvm.org/svn/llvm-project/test-suite/trunk/
not about the "test (no-dash) suite" that is part of LLVM.

If you are using the LLVM test-suite project directly (as opposed to
via LNT), I'd appreciate it if you could reply to me personally with
the following information:

1. What facilities of the project do you use:
[ X] User defined test functionality
[ X] Ability to execute two compilers (baseline, modified)
[ ] Ability to use Makefile's to modify and rerun small parts of tests
[ X] Test suite ability to work with LLVM bit code files and tools (as
opposed to native object files)
[ X] Ability to test SPEC
[ ] Ability to execute tests on a remote target
[ X] Ability to test Fortran codes

The ability to run Fortran code is something we've used in the past, and we may want to use it in the future (at least if there's Fortran code in SPEC 2006). However, I think it's a very low priority for us.

2. What compilers do you use the project to test:
[ X] clang
[ ] llvm-gcc
[ ] dragonegg
[ X] Other LLVM compilers (?)

By other compilers, I mean research prototypes. Some are modified versions of Clang; others are passes that are loaded into opt. Others might use llvm-gcc (although I don't know of any off-hand that do that we care about).

[ ] Other non-LLVM compilers

3. What TEST variants do you use:
[ ] I do not know what this means
[ ] nightly
[ ] simple
[ X] Other, please specify: _________

We use hand-written variants in SAFECode, Poolalloc, and in internal research projects. These Makefiles are out-of-tree but plug into the test suite using an extension mechanism that Chris Lattner originally wrote for the DSA/Poolalloc project.

We have used test-suite heavily in our research experiments and open-source project testing. The two features that are most beneficial to us are:

1) We can plug into the LLVM test suite to do customized experiments without modifying the core test-suite Makefiles.

2) The API doesn't change, so we continue using the same infrastructure with each new LLVM release. The SAFECode and Poolalloc tests were written years ago but have continued working with only minor modification. Many of our internal projects have just copied and modified what they've done. Experimental infrastructure is a pain to set up, so having most of the work done is a real benefit.

-- John T.

Hi Daniel,

I would like to get some feedback on who in the community is using the
LLVM test-suite repository, and in what ways.

I am talking specifically about the "test-suite" SVN project, e.g.:
   http://llvm.org/svn/llvm-project/test-suite/trunk/
not about the "test (no-dash) suite" that is part of LLVM.

If you are using the LLVM test-suite project directly (as opposed to
via LNT), I'd appreciate it if you could reply to me personally with
the following information:

I have a buildbot running the nightly testsuite. Besides that:

1. What facilities of the project do you use:
[ ] User defined test functionality
[ ] Ability to execute two compilers (baseline, modified)
[ ] Ability to use Makefile's to modify and rerun small parts of tests
[ ] Test suite ability to work with LLVM bit code files and tools (as
opposed to native object files)
[X] Ability to test SPEC
[ ] Ability to execute tests on a remote target
[X] Ability to test Fortran codes

^ OK, I've not recently used the ability to test Fortran, but I would like to
improve the testsuite to use gfortran+dragonegg for this.

2. What compilers do you use the project to test:
[ ] clang
[ ] llvm-gcc
[X] dragonegg
[ ] Other LLVM compilers (?)
[ ] Other non-LLVM compilers

3. What TEST variants do you use:
[ ] I do not know what this means
[X] nightly
[X] simple
[ ] Other, please specify: _________

Ciao, Duncan.

Hey Daniel,

My usage hasn't changed much recently, with the exception that I use 'simple' more than 'nightly' these days. I only use nightly when I want to exercise the baseline vs. modified functionality via LLC_BETA. I run pretty much exclusively via the Makefiles.

-Jim

Hi Daniel,

1. What facilities of the project do you use:
[ ] User defined test functionality
[X] Ability to execute two compilers (baseline, modified)
[X] Ability to use Makefile's to modify and rerun small parts of tests
[X] Test suite ability to work with LLVM bit code files and tools (as
opposed to native object files)
[ ] Ability to test SPEC
[ ] Ability to execute tests on a remote target
[ ] Ability to test Fortran codes

I'll generally test on a subset while doing development, and run on the full suite when I'm doing something else.

It is very likely that I'll be interested in SPEC and Fortran in the near future.

2. What compilers do you use the project to test:
[X] clang
[ ] llvm-gcc
[ ] dragonegg
[ ] Other LLVM compilers (?)
[X] Other non-LLVM compilers

By 'clang' I really mean 'opt'. I never actually modify clang.

The non-LLVM compilers part isn't quite true (yet), but I'll be using test-suite to test a custom GCC plugin at some point in the next two weeks (I'm 95% sure I can make this work, somehow). Not the intended purpose of test-suite, I know.

3. What TEST variants do you use:
[ ] I do not know what this means
[ ] nightly
[ ] simple
[X] Other, please specify: __profile__

Currently I mostly use my own TEST.profile.[Makefile,report]. This has been sent to llvm-commits and is awaiting review (I'll be resending very soon as I just had to update it due to the RUNSAFELY changes.)

Regards,
Alastair.