RFC: Adding GCC C Torture Suite to External Test Suites

TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional external source of tests for the “nightly” test suite. If you are willing to review the patch, it is here: https://reviews.llvm.org/D66887

Background:

While working on the RISC-V backend, we have found it useful to use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests, in order to ensure the compiler is correct. Internally at lowRISC, we have been running the GCC C Torture suite [1] using custom scripts in order to ensure the backend can handle these tests as well.

The main advantage is that the torture suite provide a corpus of simple executable tests that can relatively easily be minimised to identify the sources of bugs or regressions. During the bringup of the RISC-V backend, the executable torture tests were the main litmus test for backend correctness.

We are not the only backend to have found this test suite helpful, it is noted in the WebAssembly backend that they have found the GCC Torture suite helpful during their backend bring-up as well [2].

The GCC C Torture Suite is composed of four sets of tests. The set we have found most useful is the “execute” tests, which are single-source-file test cases that should be able to be compiled, linked, and run without issues. The next most useful group are the “compile” tests, which should be able to be compiled without issues. There is more documentation about the test suites here, under “gcc.c-torture” [3].

Implementation:

The LLVM Nightly test suite has support for external test suites like SPEC, where there are licencing difficulties with the tests, but where the llvm project would still like to be able to run these tests as part of our general compiler testing and benchmarking.

I have prepared a patch to llvm-test-suite [0], which adds support for the “execute” tests in the GCC C Torture Suite. At the moment, this patch only covers the CMake build system, which I am more familiar with.

Limitations:

The main limitation at the moment (beyond lack of Makefile support) is that the GCC torture suite has a lot of tests that rely on internal features of GCC, or GNU extensions that clang has not implemented (like nested functions). At the moment the patch includes a long blacklist of test cases that do not run with clang. This is sufficient for the moment, but could of course be improved - the DejaGnu comments in the test cases allow them to add target-specific C Flags, and to skip tests that rely on internal features.

A promising direction on this point is the work by some Embecosm developers on a version of the GCC Test Suite that is explicit about which test cases need GCC internal features [4]. This has included both modernising tests, and adding better DejaGnu filters and options.

In the future, it would be nice to be able to generate the list of compiler-compatible target-specific test cases (and their cflags) from the DejaGnu information, and use this information within the CMake test suite. Most optimal would be the ability to generate this info, and save the generated info into the test suite so that test runners don’t need to run an additional tool during configuration.

Summary:

I am proposing to add the GCC C Torture suite as an additional external test suite. This will allow us to run these extra test cases regularly and within existing LLVM test suite infrastructure. I welcome volunteers to review [0], and input from anyone else who has been using the torture suite in compiler development or has relevant out-of-tree work.

[0]: https://reviews.llvm.org/D66887
[1]: https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/testsuite/gcc.c-torture/
[2]: https://github.com/llvm/llvm-project/blob/master/llvm/lib/Target/WebAssembly/README.txt
[3]: https://gcc.gnu.org/onlinedocs/gccint/C-Tests.html
[4]: https://github.com/embecosm/gcc-for-llvm-testing

TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional external source of tests for the “nightly” test suite. If you are willing to review the patch, it is here: https://reviews.llvm.org/D66887

Background:

While working on the RISC-V backend, we have found it useful to use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests, in order to ensure the compiler is correct. Internally at lowRISC, we have been running the GCC C Torture suite [1] using custom scripts in order to ensure the backend can handle these tests as well.

The main advantage is that the torture suite provide a corpus of simple executable tests that can relatively easily be minimised to identify the sources of bugs or regressions. During the bringup of the RISC-V backend, the executable torture tests were the main litmus test for backend correctness.

We are not the only backend to have found this test suite helpful, it is noted in the WebAssembly backend that they have found the GCC Torture suite helpful during their backend bring-up as well [2].

The GCC C Torture Suite is composed of four sets of tests. The set we have found most useful is the “execute” tests, which are single-source-file test cases that should be able to be compiled, linked, and run without issues. The next most useful group are the “compile” tests, which should be able to be compiled without issues. There is more documentation about the test suites here, under “gcc.c-torture” [3].

Implementation:

The LLVM Nightly test suite has support for external test suites like SPEC, where there are licencing difficulties with the tests,

Given the issues with blacklisting tests, etc. I wonder whether it would
be more useful, and acceptable, to import the tests themselves. Are the
tests all, to the best of your knowledge, GPLv3?

-Hal

Yes, the tests are, by our understanding, GPLv3. Importing could be a
reasonable option - but how do you see that interacting with the need
for blacklisting tests? Are you imagining that we just wouldn't import
the GCC-only tests into the repo? Obviously there'd still be potential
architecture-specific blacklisting, and hassle around tests that need
custom options.

Best,

Alex

TL;DR: I am proposing to add the GCC C Torture suite [1], as an additional external source of tests for the “nightly” test suite. If you are willing to review the patch, it is here: https://reviews.llvm.org/D66887

Background:

While working on the RISC-V backend, we have found it useful to use additional test suites beyond the in-tree Clang and LLVM tests and the LLVM nightly tests, in order to ensure the compiler is correct. Internally at lowRISC, we have been running the GCC C Torture suite [1] using custom scripts in order to ensure the backend can handle these tests as well.

The main advantage is that the torture suite provide a corpus of simple executable tests that can relatively easily be minimised to identify the sources of bugs or regressions. During the bringup of the RISC-V backend, the executable torture tests were the main litmus test for backend correctness.

We are not the only backend to have found this test suite helpful, it is noted in the WebAssembly backend that they have found the GCC Torture suite helpful during their backend bring-up as well [2].

The GCC C Torture Suite is composed of four sets of tests. The set we have found most useful is the “execute” tests, which are single-source-file test cases that should be able to be compiled, linked, and run without issues. The next most useful group are the “compile” tests, which should be able to be compiled without issues. There is more documentation about the test suites here, under “gcc.c-torture” [3].

Implementation:

The LLVM Nightly test suite has support for external test suites like SPEC, where there are licencing difficulties with the tests,

Given the issues with blacklisting tests, etc. I wonder whether it would
be more useful, and acceptable, to import the tests themselves. Are the
tests all, to the best of your knowledge, GPLv3?

Yes, the tests are, by our understanding, GPLv3. Importing could be a
reasonable option - but how do you see that interacting with the need
for blacklisting tests? Are you imagining that we just wouldn't import
the GCC-only tests into the repo?

That was, indeed, what I had in mind, but...

  Obviously there'd still be potential
architecture-specific blacklisting, and hassle around tests that need
custom options.

In general, it seems like keeping any blacklists and special options in
sync with the tests will be easiest if the configurations and the tests
are together in the repository.

I'm just curious what your thoughts are on which will be better in the
long term.

Thanks again,

Hal

There are 1500 tests total, and about 100 on the platform-agnostic blacklist. Alex and I do not think this is an onerous burden for maintenance, either as an external test suite or if the test suite is imported.

In the long term, if we import the tests, we know we will have to do updates when the Embecosm work lands, and beyond that updates can be more sporadic. It’s not clear to me how much harder these updates will be than if the test suite remains external.

We would welcome more views as to whether this suite should be imported or should be an external test suite.

It would also be useful to understand the status of the CMake vs Makefile build system in the nightly test suite, to understand whether we need to support Makefiles, or whether Makefiles will be disappearing from the nightly tests soon (as they have already from the main project repo).

Sam

There are 1500 tests total, and about 100 on the platform-agnostic blacklist. Alex and I do not think this is an onerous burden for maintenance, either as an external test suite or if the test suite is imported.

In the long term, if we import the tests, we know we will have to do updates when the Embecosm work lands, and beyond that updates can be more sporadic. It’s not clear to me how much harder these updates will be than if the test suite remains external.

We would welcome more views as to whether this suite should be imported or should be an external test suite.

I lean toward importing - I suspect we'll get better coverage on
buildbots, and just in general more people will end up using the tests,
than if it is external. I'm also curious what other people think.

-Hal

I also thought that importing the tests will result in them being run far more regularly.
I wonder in how far regressions happen in these tests after a backend has been brought up with it. I.e. once all these tests are made to pass, do they later still capture regressions from time to time, or do they pretty much always keep on passing after?
If they do catch regressions later on, the value of running them more frequently (e.g. on buildbots) goes up quite a bit.

Maybe the only reason I could think of to not import them is if they would take a long time to run - making buildbots slower. Is there any data on how long it takes to run these tests?

Thanks,

Kristof

They're fast to build+run. Compiling and executing all ~1400
non-masked tests on an i9-9900k takes less than a minute (~40s):
* That's using a release build of Clang (will be slower for Debug+Asserts)
* Compiling tests with O2 targeting RISC-V
* Running tests using qemu-user

I typically run a whole bunch of ISA variant + opt level + ABI
variants. You'd obviously hope that regressions don't happen, but it's
a useful sanity check to complement the in-tree unit tests. It's
helped me catch things when reviewing patches from others and
developing my own.

I also tend to generate .s, check them into a git repo and use git
diff to check for unexpected changes in output.

My general feeling is that if a regression or unexpected code change
(e.g. code size regression) can be found in one of the torture suite
tests then it's great news (vs seeing tthe same problem in a larger
program) - you've already got a fairly small and easy to understand
input so it's usually pretty easy to minimise and get to the root of
the problem.

Best,

Alex

Given the replies so far have been tending towards importing rather than having the test suite be external, I have updated the patch, here: https://reviews.llvm.org/D66887

I would welcome reviewers, noting that the patch is huge, and really the important parts for this review are the build scripts.

I have not added Makefile support for these tests. I still don’t know the status of the Makefile build system - whether it’s staying around permanently, or whether it’s due to be removed like it was from the main llvm repos.

Sam

Thanks to the help of Kristof, Hal, Alex, Lewis Revill, and Henry Cheang, the GCC C Torture suite is now part of the llvm test suite.

In the end, we added the source files to the repo rather than having it as an external test suite.

The relevant commits are:
- https://reviews.llvm.org/rT374155 (Adding the CMake Configuration and licences).
- https://reviews.llvm.org/rT374156 (Adding the Test Case source files).

There are two things to note:
1. Currently we only enable the gcc c torture suite for x86 and riscv architectures. This is to avoid breakage on other architectures, as we’re using blacklists to exclude tests.
2. There is only a CMake configuration for the gcc c torture suite. I have not added Makefile support.

We would welcome the addition of support for other architectures. I was not able to do this as I do not have a testing setup for more than the two architectures above.

Sam