Parallelizing clang tests

I have access to a quad Mac Pro now and have started building clang in the llvm tree.

I know that clang's test system is independent of dejagnu (which is a good thing).
While TestRunner.sh is pretty basic it does a good job.

But I expect that the clang tests will multiply soon. So I thought of
using extra cores to speed up tests.

The idea is this:

typing "make test" could proceed as follows:

0) cd into the test/ directory and make all there, which in turn
1) collects *.m, *.c, *.cpp files and creates targets *.cpp.testresults
     into a fresh makefile: Makefile.tests
2) in Makefile.tests some administrative targets like "all", "report", "clean"
    are established.
3) Makefile.tests is run with targets "run" and "report".

*.testresults are built by executing ./TestRunner.sh and capturing all output

the "report" targes just catenates all *.testresults to stdout.

So the behaviour is not grossly different, just the errors are shown
at the *end*.

The big profit of this approach is that specifying -j 4 to make
lets the testing proceed in parallel!

Below you find the new makefile, it is not yet polished, but works.
Please tell me whether I can proceed an commit.

Some empirical results.

My timings for the 4-core machine indicate that

"make test" takes 23 seconds.
"make -j 2 test" takes 15 seconds
"make -j 4 test" takes 13 seconds

A significant speedup I would say.

Cheers,

  Gabor

This is really cool Gabor. I'd really like to see this. One additional request: in terms of output, it is nice to be able to get output as tests are run. This lets you know if the test harness has hung and lets you see what progress it is making. However, most tests are successful, so seriously verbose output is bad (and potentially confusing when -j is used).

How about this approach:

1. by default, a successful test, immediately after it is run, prints out a single "." with no newline.
2. by default, a failing test prints out "\n---- foo.c failed ---\n" or whatever.
3. When VERBOSE=1 is passed to make, it could print all the tests it is running like it does today: print the test name before it is executed, which is useful if the test hangs.

This means that we'll get a nice "progress bar" out of dots in the common case when everything is passing, but that the order of the dots won't matter, and if individual dots get intermixed because of -j, it won't matter.

What do you think?

-Chris

I know this is a little off-topic, but would anyone be opposed to a "valgrind mode" for "make test"? ("make test VG=1", probably.)

If so, is there a valgrind-like tool for the Mac? It looks like a Darwin/x86 port is high priority for the valgrind developers, but currently not available.

I would be happy to add support for this, if you all think it is a good idea. Or Gabor is welcome to.

Sam

Chris Lattner wrote:

I know this is a little off-topic, but would anyone be opposed to a
"valgrind mode" for "make test"? ("make test VG=1", probably.)

That would also be very very nice. Valgrind is a great tool.

If so, is there a valgrind-like tool for the Mac? It looks like a
Darwin/x86 port is high priority for the valgrind developers, but
currently not available.

Not yet :(.

I would be happy to add support for this, if you all think it is a
good idea. Or Gabor is welcome to.

I think it would be a great idea. Valgrind is very useful for tracking down some of the most evil bugs :slight_smile:

-Chris

Just for the record, I have checked this in as test/Makefile.parallel,
you need this patch to clang/Makefile to activate it:

Index: Makefile

I know this is a little off-topic, but would anyone be opposed to a
"valgrind mode" for "make test"? ("make test VG=1", probably.)

If so, is there a valgrind-like tool for the Mac? It looks like a
Darwin/x86 port is high priority for the valgrind developers, but
currently not available.

Didn't Apple employee Greg Parker claim that he had a working port of valgrind that he'd release as soon as Leopard is out? Do you Apple guys know what happened to that?

Nico

AFAIK, it is working with some caveats. I'm not sure the current status, but I know he's been busy lately. I'd suggest sending him a polite email asking what his plans are.

-Chris

Very nifty. Some "polish":

a) it would be nice for 'make test VERBOSE=1' to print the old output (each test name as it is run)
b) when a test fails, it appears that the failure notice is queued up and printed when testing is done. It would be nice to print those as discovered.
c) really trivial, it would be nice to print "Running tests: " before the .'s start flying out.

Overall this is very cool.

-Chris