Help with RISCV and QEMU in llvm testsuite lit testing

Hi,

I am not sure if this is the right forum. Please direct me to the appropriate place if it isn’t so.

Please keep in CC as i am not subscribed to this mailing list.

I am trying to test riscv llvm tools in QEMU using llvm testsuite. As a trial i am trying only the Single Source C Regression folder. The steps that i took are :

]$ cmake -DCMAKE_C_COMPILER=/opt/riscv-tools/bin/clang -DCMAKE_C_FLAGS="–target=riscv32 -march=rv32i -mabi=ilp32" -DCMAKE_CXX_COMPILER=/opt/riscv-tools/bin/clang -DCMAKE_CXX_FLAGS="–target=riscv32 -march=rv32i -mabi=ilp32" -DCMAKE_EXE_LINKER_FLAGS="-lm" -DTEST_SUITE_RUN_UNDER=/opt/qemu/bin/qemu-riscv32 -DTEST_SUITE_COLLECT_STATS=OFF -DTEST_SUITE_USER_MODE_EMULATION=ON -DTEST_SUITE_USE_PERF=ON -DARCH=“riscv” -DTEST_SUITE_SUBDIRS=“SingleSource/Regression/C” -C…/llvm-test-suite/cmake/caches/O3.cmake …/llvm-test-suite

]$ make -j 12 VERBOSE=1

]$ /opt/riscv-tools/bin/llvm-lit -j 12 -o results.json .

I see that all the tests are failing. When i try the tests manually i could see that results produced are same as that of the reference output. When verbose mode is enabled i see that *.test.out files are not being produced. This is the file that gets compared with the reference file.

FAIL: test-suite :: SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer.test (1 of 35)
******************** TEST ‘test-suite :: SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer.test’ FAILED ********************

/òpt/llvm-test/build/tools/timeit --limit-core 0 --limit-cpu 7200 --timeout 7200 --limit-file-size 104857600 --limit-rss-size 838860800 --redirect-input /dev/null --chdir /òpt/llvm-test/build/SingleSource/Regression/C --summary /òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.time /opt/riscv/install-qemu/bin/qemu-riscv32 /òpt/llvm-test/build/SingleSource/Regression/C/Regression-C-2003-06-16-InvalidInitializer
cd /òpt/llvm-test/build/SingleSource/Regression/C ; /òpt/llvm-test/build/tools/fpcmp /òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out 2003-06-16-InvalidInitializer.reference_output

  • cd /òpt/llvm-test/build/SingleSource/Regression/C
  • /òpt/llvm-test/build/tools/fpcmp /òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out 2003-06-16-InvalidInitializer.reference_output
    /òpt/llvm-test/build/tools/fpcmp: error: unable to open ‘/òpt/llvm-test/build/SingleSource/Regression/C/Output/Regression-C-2003-06-16-InvalidInitializer.test.out’

I am not sure who is responsible for the *.test.out file. I am assuming that this should be generated by QEMU. But I see that there is “exit 0” in the reference files that are not produced in the QEMU output. Should qemu be run via a wrapper that captures the output and “exit 0” in a *.test.out file ?

Any help is appreciated.

Thanks for your time.

Regards,
Shafi

The "exit 0" should be produced by the tools/timeit.sh (since you are
using -DTEST_SUITE_USE_PERF=ON) and has a cmdline switch
--append-exitstatus. The command line for it is produced by
litsupport/modules/timeit.py. No idea why it does not work in your
case, but I hope this help for looking into the issue.

Michael

Thanks for the response. What you have pointed out is correct. Since i
have enabled TEST_SUITE_USER_MODE_EMULATION the comparison with the
reference file is done with a *.out file rather than with the
simulator output directly. The appropriate out file with the simulator
output and the exit code will get generated if timeit.sh is invoked
with --append-exitstatus and --redirect-output options. timeit.sh gets
invoked by the timeit.py file with these options when
config.traditional_output is true. Interestingly lit.local.cfg in
test-suite\SingleSource has this enabled. But for some reason which i
still haven't found out, timeit.sh is invoked as though
config.traditional_output is false. From what i see this issue should
crop up when RISCV target (or a QEMU target for that matter) is tested
with llvm test suite. It would be of great help if some one could
point out what i am doing wrong.

Regards,
Shafi

Hi!

At lowRISC, we are running the llvm test suite for RISC-V using qemu.

I note two major differences between how we run our test-suite builds, and how you are running yours.

The first is that instead of running cmake and then lit, we use lnt, which will run cmake and then lit for us. The advantage of LNT is that it reports when compilation fails as well, in a unified format.

The other difference is that we don’t use `TEST_SUITE_USE_PERF`. I don’t know how this interacts with the use of qemu. I would suggest starting by turning this off, and seeing if that fixes your problem. At that point you’ll know whether the problem lies in the timeit.sh script, or somewhere else.

Micheal Kruse has pointed to where the “.out” files come from. To dig a little deeper: the build system creates a “.test" file for lit. This is then modified by some of the modules in litsupport/modules (the list of modules that modify the tests are in the `LIT_MODULES` cmake variable). The important one is `timeit.py`, which wraps the executable invocation (excluding cd) in an invocation of `timeit` which records how long the script takes etc. Timeit is responsible for redirecting output and recording exit status.

You are correct that the value of `config.traditional_output` is set in a `lit.local.cfg` in `SingleSource`. This should be being enabled, but maybe this is being skipped? It might be worth trying to print from this cfg file (it’s just python) and seeing if anything gets printed or not.

I don’t believe that `TEST_SUITE_USER_MODE_EMULATION` changes what is compared with what. The output is always saved to a file and that file is compared with a reference output file. The difference is just that we need to build `fpcmp` so that it can be executed on the host, rather than from inside qemu (the target).

I hope this helps you to debug your issue. As I said, a good first step is to turn off using PERF, and go from there.

Sam