There was a discussion about Fujitsu test suite in the last community call. IIUC, the main point was the integration into llvm-test-suite. I’m afraid I don’t perfectly understand what the blockers are. In fact, we are preparing some patches to support running only Fortran tests and reduce disk usage, and they will be merged in the near future. However, I’m not sure these are enough for the integration.
I’ve created a topic to discuss this in writing.
If I remember correctly, the primary request was to add the ability to run a subset of the tests, in particular the OpenMP tests.There was a question about whether the Fujitsu Test Suite could be added to the llvm-test-suite. It was pointed out that it (the Fujitsu Test Suite) already uses the mechanisms provided by the llvm-test-suite anyway, so that should not be difficult.
The main issue that comes to my mind, aside from the cost, is that the llvm-test-suite is expected to pass by default. Therefore, if any tests in the Fujitsu Test Suite are known to fail - perhaps because of unimplemented features in flang - there should be a mechanism to disable them.
With the gfortran test suite that has been imported into the llvm-test-suite, these disabled tests have to be manually re-enabled. This is not ideal, but it does prevent buildbot failures if someone implements a feature but forgets to update the test suite.
I am one of the developers working on improving the Fujitsu Test Suite.
Thank you for bringing this up in the community call and for your comments.
We have received requests, including via issues, to support running only the OpenMP tests, and we plan to address this. As a first step, we are planning to introduce CMake options in the February release to allow selecting tests by language (C/C++/Fortran).
Also, while the current Fujitsu Test Suite has many failing tests, most of them are due to misclassification of numerical accuracy differences as failures. Since many of these can be improved by relaxing tolerances or adjusting options, we are prioritizing this work. We expect that the majority of these failures will be resolved step by step in the releases around February and April.