Hi All,
I am working on a bug in clang. I already have a fix for it and I am going
through the "LLVM Testing Infrastructure Guide" to make sure I haven't
broken anything else. I have few questions regarding the test suite
infrastructure that I hope someone can answer.
1. I checked out and built llvm, clang, and test-suite from svn tip. When I
run llvm/test *without* my changes on x86, I see 3 failures. Is it expected?
I was expecting svn tip on x86 to be clean. Is there a nightly regression
result available somewhere that I can check for broken tests. Here is the
report from my log:
Failing Tests (3):
Clang :: Analysis/iterators.cpp
Clang :: Driver/hello.c
Clang :: Preprocessor/macro_paste_c_block_comment.c
Expected Passes : 8757
Expected Failures : 69
Unsupported Tests : 551
Unexpected Failures: 3
2. I see clang has its own test suite under clang/test. I assume when I run
llvm/test, the test infrastructure also runs clang/test.
3. For running projects/test-suite, the instruction guide directs to build
llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
4. I see many compile errors when running projects/test-suite. Is it
expected?
5. I used "make -i" to ignore compile erros and execute all tests under
projects/test-suite. I finally get a report something like this:
Program |
GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC CBE
LLC LLC-BETA JIT | GCC/CBE GCC/LLC
GMultiSource/Applications/Burg/burg |
5.5484 115468 2.9722 * 3.0922 | 0.0100 *
0.0000 * 3.2400 | n/a - n
MultiSource/Applications/ClamAV/clamscan |
41.8306 1335184 34.4382 * 18.6692 | 0.1200 *
0.1300 * 19.5200 | n/a 0.92 n
I was also expecting to see a list of passing/failing test cases similar to
llvm/test runs. Is there a make target to do so?
I apologies for asking so many questions. I followed directions from the
"LLVM Testing Infrastructure Guide" but it doesn't address all these issues.
I will appreciate your help to resolve these.
Thanks,
Sundeep
Hi All,
I am working on a bug in clang. I already have a fix for it and I am going
through the "LLVM Testing Infrastructure Guide" to make sure I haven't
broken anything else. I have few questions regarding the test suite
infrastructure that I hope someone can answer.
1. I checked out and built llvm, clang, and test-suite from svn tip. When I
run llvm/test *without* my changes on x86, I see 3 failures. Is it expected?
I was expecting svn tip on x86 to be clean. Is there a nightly regression
result available somewhere that I can check for broken tests. Here is the
report from my log:
Failing Tests (3):
Clang :: Analysis/iterators.cpp
Clang :: Driver/hello.c
Clang :: Preprocessor/macro_paste_c_block_comment.c
Expected Passes : 8757
Expected Failures : 69
Unsupported Tests : 551
Unexpected Failures: 3
Because you're living life on the bleeding edge (i.e., pulling code from the svn repository) it's possible for this to happen from time to time. A fix is probably already in the works. One way to ensure you don't introduce a new error is to maintain a clean (i.e., unmodified version of llvm/clang), which you can use as a baseline. Run regressions on both the clean and modified versions to see if you cause any additional failures.
2. I see clang has its own test suite under clang/test. I assume when I run
llvm/test, the test infrastructure also runs clang/test.
How are you invoking the tests? I typically do 'make check-all' from my build directory, which runs both regression suites.
3. For running projects/test-suite, the instruction guide directs to build
llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
Very possible the documentation needs updating.
4. I see many compile errors when running projects/test-suite. Is it
expected?
No.
5. I used "make -i" to ignore compile erros and execute all tests under
projects/test-suite. I finally get a report something like this:
Program |
GCCAS Bytecode LLC compile LLC-BETA compile JIT codegen | GCC CBE
LLC LLC-BETA JIT | GCC/CBE GCC/LLC
GMultiSource/Applications/Burg/burg |
5.5484 115468 2.9722 * 3.0922 | 0.0100 *
0.0000 * 3.2400 | n/a - n
MultiSource/Applications/ClamAV/clamscan |
41.8306 1335184 34.4382 * 18.6692 | 0.1200 *
0.1300 * 19.5200 | n/a 0.92 n
I was also expecting to see a list of passing/failing test cases similar to
llvm/test runs. Is there a make target to do so?
You might try 'make -i report'
Chad
> 1. I checked out and built llvm, clang, and test-suite from svn tip.
When I
> run llvm/test *without* my changes on x86, I see 3 failures. Is it
expected?
> I was expecting svn tip on x86 to be clean. Is there a nightly
regression
> result available somewhere that I can check for broken tests. Here is
the
> report from my log:
>
> Failing Tests (3):
> Clang :: Analysis/iterators.cpp
> Clang :: Driver/hello.c
> Clang :: Preprocessor/macro_paste_c_block_comment.c
>
> Expected Passes : 8757
> Expected Failures : 69
> Unsupported Tests : 551
> Unexpected Failures: 3
>
Because you're living life on the bleeding edge (i.e., pulling code
from the svn repository) it's possible for this to happen from time to
time. A fix is probably already in the works. One way to ensure you
don't introduce a new error is to maintain a clean (i.e., unmodified
version of llvm/clang), which you can use as a baseline. Run
regressions on both the clean and modified versions to see if you cause
any additional failures.
Makes sense.
> 2. I see clang has its own test suite under clang/test. I assume when
I run
> llvm/test, the test infrastructure also runs clang/test.
How are you invoking the tests? I typically do 'make check-all' from
my build directory, which runs both regression suites.
Yes, I am doing "make check-all".
> 3. For running projects/test-suite, the instruction guide directs to
build
> llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
Very possible the documentation needs updating.
I see. Do you run projects/test-suite with clang?
> 4. I see many compile errors when running projects/test-suite. Is it
> expected?
No.
I will try to debug. Do you have a log file from your previous runs that I
can use?
Thanks Chad. I appreciate it.
-Sundeep
1. I checked out and built llvm, clang, and test-suite from svn tip.
When I
run llvm/test *without* my changes on x86, I see 3 failures. Is it
expected?
I was expecting svn tip on x86 to be clean. Is there a nightly
regression
result available somewhere that I can check for broken tests. Here is
the
report from my log:
Failing Tests (3):
Clang :: Analysis/iterators.cpp
Clang :: Driver/hello.c
Clang :: Preprocessor/macro_paste_c_block_comment.c
Expected Passes : 8757
Expected Failures : 69
Unsupported Tests : 551
Unexpected Failures: 3
Because you're living life on the bleeding edge (i.e., pulling code
from the svn repository) it's possible for this to happen from time to
time. A fix is probably already in the works. One way to ensure you
don't introduce a new error is to maintain a clean (i.e., unmodified
version of llvm/clang), which you can use as a baseline. Run
regressions on both the clean and modified versions to see if you cause
any additional failures.
Makes sense.
2. I see clang has its own test suite under clang/test. I assume when
I run
llvm/test, the test infrastructure also runs clang/test.
How are you invoking the tests? I typically do 'make check-all' from
my build directory, which runs both regression suites.
Yes, I am doing "make check-all".
3. For running projects/test-suite, the instruction guide directs to
build
llvm-gcc. Why is llvm-gcc required? Why is clang not being used?
Very possible the documentation needs updating.
I see. Do you run projects/test-suite with clang?
From your build/project/test-suite directory, try running:
make ARCH=x86_64 CC_UNDER_TEST_IS_CLANG=1 CC_UNDER_TEST_TARGET_IS_X86_64=1 DISABLE_CBE=1 DISABLE_JIT=1 ENABLE_HASHED_PROGRAM_OUTPUT=1 ENABLE_OPTIMIZED=1 LD_ENV_OVERRIDES="env DYLD_LIBRARY_PATH=/path/to/your/build/Release+Asserts/lib" LLC_OPTFLAGS=-O3 LLI_OPTFLAGS=-O3 OPTFLAGS=-O3 SMALL_PROBLEM_SIZE=1 TARGET_CC="" TARGET_CXX="" TARGET_FLAGS="-arch x86_64" TARGET_LLVMGCC=/path/to/your/build/Release+Asserts/bin/clang TARGET_LLVMGXX=/path/to/your/build/Release+Asserts/bin/clang++ TEST=simple USE_REFERENCE_OUTPUT=1 report
***Note the ENABLE_OPTIMIZED=1 flag requires that you run configure with '--enable-optimized' at config time (i.e., ../llvm/configure --enable-optimized).
You may have to fiddle with the options, but I think this will get you going in the right direction.
4. I see many compile errors when running projects/test-suite. Is it
expected?
No.
I will try to debug. Do you have a log file from your previous runs that I
can use?
I don't have any on hand, nor do I think they would be of help. Try the above make command a see what happens. For the most part everything should PASS.
Thanks Chad. I appreciate it.
You're welcome Sundeep.
Hi Chad,
I finally got it to work. I think "LLVM Testing Infrastructure Guide" should
be updated to reflect the latest changes (use clang instead of llvm-gcc and
make command etc.).
I ran into problems with using system headers and startup files (crt*.o). So
I had to build and install llvm-gcc, next build clang using headers from
llvm-gcc install dir, and then use clang to run regressions under
projects/test-suite. It works beautifully.
Anyways, I wanted to thank you for all your help. I really appreciate it.
-Sundeep