I’m getting started with writing tests for flang and have a few questions. First, would you recommend contributing tests to the llvm-test-suite repository at https://github.com/llvm/llvm-test-suite?
Second, I was informed in April that the f18-llvm-project fork’s fir-dev branch is able to produce executable files from Fortran 77 source. Could someone explain how to use this capability and whether this capability has been merged upstream to the llvm-project repository? I can see from an error message that my locally installed flang is invoking gfortran to produce executable files. Do I need to pass a flag to tell flang to produce an executable program without invoking gfortran?
Yes, I think the llvm-test-suite should feature more Fortran tests.
See the documentation at [1] and [2] as an example for how tests are
added. For instance, NASA Advanced Supercomputing Parallel Benchmarks
[3] could be added.
Second, I was informed in April that the f18-llvm-project fork's fir-dev branch is able to produce executable files from Fortran 77 source.
Could someone explain how to use this capability and whether this capability has been merged upstream to the llvm-project repository?
Merging this upstream is an ongoing process. I am not active in this area so can't comment on timescales, but it is one of our main challenges right now.
As for compiling with the fir-dev branch on f18-llvm-project, we have been recently discussing this on flang-compiler.slack.com. Are you active there? I can send you an invite.
Do I need to pass a flag to tell flang to produce an executable program without invoking gfortran?
`flang` that you are referring to is just a bash wrapper script [1] that was added for experimenting. It will only use LLVM Flang to parse-unparse the input source, and then will call e.g. `gfortran` for code-generation. You can replace `gfortran` with some other Fortran compiler. The name is rather confusing. We should rename it as `flang-to-gnu` or something similar.
If you intend to use the fir-dev branch. You may find the following instructions helpful.
You can build Flang on the fir-dev branch by enabling the flang project along with clang, mlir and openmp.
Note: Please change TARGETS_TO_BUILD to your platform (X86, PowerPC).
It is possible that the linking process might complain about some math symbols. If so, you must download classic flang and place the library libpgmath.so somewhere and point to it in the link step.
And then you can find lipgmath.* in the lib directory in build.
(There might be an option to build pgmath automatically if you set an environment variable. There might also be an option to use mlir/libm math functions instead of pgmath. But I have not explored these options.)
We recently added a (temporary?) directory for intrinsic tests under “flang/test/Intrinsics” but we still need to decide how to best integrate it into the testing infrastructure. For now, my vote would be that it is OK to land your tests there and we can then decide where these tests should live in the long run. Personally, I think the intrinsic specific details (both lowering and “correctness”) make a bit more sense as part flang’s testing vs. rolling them all the way up to the llvm level. But that’s just my take — others should chime in if there is a better path…
I agree with Pat. For me, anything that tests one of Flang's internal representations (parse-tree, FIR, LLVM IR) or the generated diagnostics, belongs somewhere within flang/test (i.e. with other regression tests). I always run all of them (ninja check-flang) when developing a patch. I rarely use the test-suite. Not sure about the others. Btw, LLVM's test-suite requires CMake support, which we don't have. But perhaps I'm missing something here!
As for the actual location for new tests, I've just submitted a patch to move all of regression tests for lowering the intrinsics into f18-llvm-project/flang/test/Lower/intrinsic-procedures [1]. These tests only look at FIR. We may want a different directory for LLVM IR. It would be very nice to keep all tests for intrinsics in sync
I believe, Pat and Damian are talking about execution tests for Fortran intrinsics. While these are not testing the intermediate representations, these are unit-tests that only apply to Flang. And when we make changes to the compiler, it will be good to test (with check-flang) that they pass. I don’t know what is the framework that is being chosen for running these tests, but for the trivial cases, we do not need CMake. There are already end-to-end tests running in the fir-dev branch. https://github.com/flang-compiler/f18-llvm-project/blob/fir-dev/flang/test/Lower/end-to-end-character-assignment.f90
I believe, Pat and Damian are talking about execution tests for Fortran intrinsics.
Correct. As someone who thinks more from the perspective of an application developer, I’d like to write execution tests, which is also what led me to the fir-dev branch as I don’t think the ability to produce executable files has been accepted upstream yet.
While these are not testing the intermediate representations, these are unit-tests that only apply to Flang. And when we make changes to the compiler, it will be good to test (with check-flang) that they pass. I don’t know what is the framework that is being chosen for running these tests, but for the trivial cases, we do not need CMake.
For testing, I was originally planning to use the Vegetables unit testing framework, which I like because it supports writing tests in the form of a specification. I would also like to add a capability for building the tests using the Fortran Package Manager (fpm). Doing so is trivial – much simpler than modifying the CMake and with the benefit that it’s trivial to set things up so that fpm automatically downloads any test dependencies such as Vegetables and we don’t have to touch the CMake scripts. Alternatively, if this proves problematic in any way, I’ll commit a simple assertion utility borrowed from the Sourcery library.
Just a quick note to give you a feel for the testing infrastructure here — it is much broader than what it might appear at first glance as you have to look beyond Flang.
The testing mechanisms need to be usable/stable across the community as testing and releases are done LLVM-wide (across all projects: Clang, MLIR, Flang, LLVM, etc.). As a result, any new external package dependencies could have an impact across established mechanisms across all projects (e.g., see https://lab.llvm.org/buildbot for a glance at coverage, platforms, etc.).
For this reason, I would suggest adopting existing capabilities from the current mechanisms used across the LLVM infrastructure as your starting point. A key step will be to make sure any approach taken can be run without additional requirements on the broader community — or we will have to open a discussion with the broader community about expanding the supported infrastructure. Like everything, there are always pros and cons, but keeping Flang’s approach usable and sound as a component of the larger code base is paramount to maintaining LLVM’s established testing and release process.
