Running gcc tests using Clang

Project Infrastructure LLVM Dev List Archives
1

I’ve run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu x86-64 Linux box with the following results:

=== gcc Summary ===

of expected passes 29946

of unexpected failures 9938

of unexpected successes 29

of expected failures 28

of unresolved testcases 1451

of untested testcases 273

of unsupported tests 811

pid is 4456 -4456
output is clang version 2.9 (trunk 113841)
Target: x86_64-unknown-linux-gnu
Thread model: posix
status 0
/home/john/llvm/Release+Asserts/bin/clang version 2.9 (trunk 113841)

This seems to be an improvement over the results I had previously when using the i386 configuration. However, it sounded like you had more success on a Darwin box, with only 1 unexpected failure (mine, which should be fixed now). Should I expect that with this configuration?

I found that I had to exit most other running applications to be able to run the tests without running out of memory.

I’ve enclosed my hacked makefile, hacked to change the configuration and add the -v (verbose) flag.

I have a 6 mb log file from this run if needed.

~/gcc-4_2-testsuite$ find . -print | wc
36392 36392 1687901

Some output from the first few failed tests:

Testing gcc.apple/5490617.c
doing compile
pid is 17502 -17502
output is status 0
FAIL: gcc.apple/5490617.c scan-assembler .lcomm __my_endbss

Testing gcc.apple/4104248.c
doing compile
pid is 17445 -17445
close result is 17445 exp8 0 1
output is ilp3217400.c:2: error: array size is negative
status 1

Testing gcc.apple/4641942.c
doing compile
pid is 17481 -17481
output is status 0
FAIL: gcc.apple/4641942.c (test for warnings, line 20)

Testing gcc.apple/5597292.c
doing compile
pid is 17507 -17507
close result is 17507 exp8 0 1
output is /home/john/gcc-4_2-testsuite/src/gcc.apple/5597292.c:7: error: fields must have a constant size: ‘variable length array in structure’ extension will never be supported
status 1
compiler exited with status 1
FAIL: gcc.apple/5597292.c (test for excess errors)

Testing gcc.apple/6286881.c
doing compile
pid is 17572 -17572
close result is 17572 exp8 0 1
output is /home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: warning: ^ has lower precedence than !=; != will be evaluated first [-Wparentheses]
/home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place parentheses around the ^ expression to evaluate it first
/home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: note: place parentheses around the != expression to silence this warning
/home/john/gcc-4_2-testsuite/src/gcc.apple/6286881.c:8: error: address expression must be an lvalue or a function designator
status 1
compiler exited with status 1
FAIL: gcc.apple/6286881.c (test for errors, line 8)

etc.

-John

Makefile (2.8 KB)

2

For reference, here are the results from the last time I ran it with llvm-gcc, about a month ago:

of expected passes 39621

of unexpected failures 1112

of unexpected successes 2

of expected failures 59

of unresolved testcases 80

of untested testcases 280

of unsupported tests 1073

Many of the 9000 extra failures are probably llvm-gcc/clang differences that are nothing to do with you, though. The best way to get useful info out of this is to run it with and without the patch you’re testing, and compare.

3

btw, the main log file that you’re quoting from does not generally have enough info to reproduce failures. There is a more detailed log at /gcc/testsuite/gcc/gcc.log . Beware, it is destroyed every time you rerun the test suite.

4

Hi John,

I've run the tests from clang-tests/trunk/gcc-4_2-testsuite on a Ubuntu
x86-64 Linux box with the following results:

    === gcc Summary ===

# of expected passes 29946
# of unexpected failures 9938
# of unexpected successes 29
# of expected failures 28
# of unresolved testcases 1451
# of untested testcases 273
# of unsupported tests 811
pid is 4456 -4456
output is clang version 2.9 (trunk 113841)
Target: x86_64-unknown-linux-gnu
Thread model: posix
status 0
/home/john/llvm/Release+Asserts/bin/clang version 2.9 (trunk 113841)

This seems to be an improvement over the results I had previously when using
the i386 configuration. However, it sounded like you had more success on a
Darwin box, with only 1 unexpected failure (mine, which should be fixed
now). Should I expect that with this configuration?

No, that is unfortunately not the case.

What I do for testing is keep arounds results (see the 'make
save-results' utility target) and compare the diffs from run to run.
This is a real pain, but the best tool we have at the moment.

- Daniel

5

All,

Here's a bit of old shell script I've used for comparing test results from various GCC testsuite runs. It's nothing fancy, but does cut down a bit of the noise and makes it a bit more convenient. Just point it at two directories containing the *.sum result files. YMMV and all that.

compare_test_results.sh (1.44 KB)