I have been working on adding the gfortran tests to LLVM’s test suite. I have largely followed the model used to add GCC’s C torture tests to the test suite.
There is a patch here (D146485) that adds the CMake configuration and license files to the repository. That is the primary patch that needs review. At the end of the post are links to other patches that add the test files themselves to the repository. All the patches will be committed together.
There are two broad categories of tests in the suite - “compile” tests and “execute” tests. The “compile” tests essentially test the compiler’s ability to print appropriate error messages, interact well the gdb, generate appropriate profiling information etc. The execute tests check the runtime behavior of the compiled code.
Currently, only the “execute” tests are built and run.
Of these, a number of tests have been disabled because they cannot be built. Some fail to build because they use features not yet implemented in flang and error message says as much. Others may also be a result of unimplemented features but that is not apparent from the error message. Some may be failing because they use unsupported (non-standard) extensions.
Q. Should these tests be left disabled in the test suite? In that case, the suite will build, but the list of disabled tests will have to be revisited periodically when the features that are causing their failure are implemented.
Of the tests that build, many fail to pass. I have not checked the reason for the failures. It may be that they are spurious, but they could also be legitimate failures.
Q. Should the currently failing tests also be disabled?
It may be useful to also enable the “compile” tests. Naturally, they expect GCC-specific error messages. As a temporary measure, we could just mark the relevant tests as “expected to fail” and we would at least have a set of “pass/fail” tests.
Q. From a cursory glance at the LLVM test-suite, it doesn’t seem to be geared to operate in this way i.e. it seems to be primarily intended to test the runtime performance/correctness of the compiler. If this is not the case, I could use help from someone more familiar with the test-suite to enable the compile tests at least in pass/fail mode.
Longer term, it may be useful to actually check for the appropriate error messages. This would involve replacing the DejaGNU annotations in the tests with LLVM-specific ones.
The patches that add the actual test files into the repo are here:
[EDIT]: Fixed links