Flang + CMake

Hi,

I’ve been asked by Steve to take a look at Flang CMake integration as I have upstreamed the original flang integration in the past.

I’ve taken a look and confirm that if we change the predefines that CMake looks for from upper case __FLANG to flang (and other similar predefines) cmake will be able to identify and use flang.

The question I want to pose is: what should we name this flang for cmake identification? There is already an existing flang identification in CMake that I assume people depend on. So we will need to duplicate the CMake identification bits for original flang into a different name. I would like suggestions for what to name it. (Perhaps llvm-flang?)

I can set up the initial CMake identification for flang and create a pull request upstream, but going forward someone with more familiarity with flang’s options should maintain it.

Tin

Hi Tin,

Thank you for taking a look at this!

* LLVM Flang - https://github.com/llvm/llvm-project/tree/main/flang
* Classic Flang - https://github.com/flang-compiler/flang

IIUC, the existing identification in CMake is for "Classic Flang". We want to preserve it - there will be a lot of developers that rely on it. So for "LLVM Flang", we probably want to add something new rather then modify the existing identification.

I'm not too concerned about the actual name, but I think that it would be very beneficial if we abandoned the name "Flang" in favour of something that uniquely identifies "Classic Flang" and "LLVM Flang". Otherwise this gets too confusing.

In my original e-mail I was only referring to "LLVM Flang". It was my incorrect assumption that the existing identification in CMake was introduced for "LLVM Flang". Apologies for that.

-Andrzej

FYI, I've just opened an issue for this: LLVM Flang - support (#22387) · Issues · CMake / CMake · GitLab

-Andrzej

Hi everyone,

We are a bit stuck here. From the discussions so far, it seems that we may need to make some changes to the preprocessor in order to progress this. Would someone familiar with the design of the preprocessor be able to chime in (either here, on Slack, CMake [1] or Bugzilla [2])? That would be greatly appreciated!

Lack of CMake integration is already limiting us in terms of _what_ can be tested. The project will benefit greatly if we manage to get this introduced soon. And it will make LLVM Flang much more attractive to end-users.

Best regards,
-Andrzej

[1] LLVM Flang - support (#22387) · Issues · CMake / CMake · GitLab
[2] 50993 – flang -E breaks fixed-format files

For the Ninja CMake generator, CMake uses an intrusive mechanism for finding the dependencies (through modules) among Fortran files. CMake uses the compiler to preprocess the file and then finds the dependencies and then passes the preprocessed files onto another invocation of the compiler to compile. This has tripped up a few compilers like Cray, IBM etc which they fixed recently (see links to issues below). The issue does not apply for the Makefile generator since it internally uses a Fortran parser/preprocessor (makedepf90) to go through the files and find the dependencies. See the following link for more details. An ideal non-intrusive mechanism would have been to ask the compiler to list modules used and modules defined and then build the dependencies from that.
https://mathstuf.fedorapeople.org/fortran-modules/fortran-modules.html
https://gitlab.kitware.com/cmake/cmake/-/issues/20697

https://gitlab.kitware.com/cmake/cmake/-/issues/20731

I believe in Flang, the preprocessor and prescanner (https://github.com/llvm/llvm-project/blob/main/flang/docs/Parsing.md#prescanning-and-preprocessing) run together to produce a cooked character stream (normalized with no spaces, lowercase etc). I guess prescanner helps the preprocessor to be Fortran aware. In classic flang we have had issues where we had to make the preprocessor aware of Fortran syntax (https://github.com/flang-compiler/flang/pull/658). The cooked character stream is currently the output if we use the -E flag. Due to loss of whitespace etc this is not fixed form anymore. Also, flang would have constructed the provenance information while creating the cooked character stream. A new invocation of Flang on the cooked character stream will not be able to capture the source provenance information. As Bryan points out (https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_986516) I guess line directives have to be inserted to retain this information.

Is it possible to retain fixed-form formatting after prescanning? And can we normalize it as the first step in parsing, so that the parser remains unchanged?
Can line directive information be added to retain source location information? And can this be consumed and interpreted before parsing?
Can CMake use the make generator flow? (The CMake folks have a preference to fix this in the compiler side. https://gitlab.kitware.com/cmake/cmake/-/issues/22387#note_983394.)

–Kiran

@Kiran, thank you for that great summary!

We also discussed this with Peter Klausler (Nvidia, one of the authors of the preprocessor/prescanner in LLVM Flang) on flang-compiler.slack.com. Please check the thread started by Johannes Doerfert on Monday, July 12th 2020.

The conversation on Slack is publicly available, but perhaps it is worth to summarise it here. As Kiran already highlighted, the output generated with `-E` is the "cooked character stream" rather than "pre-processed" file. In general, it's not valid Fortran source and it should be re-formatted/post-processed first. It is unfortunate that we use `-E` to generate this - perhaps we should rename this option? We did discuss the possibility of modifying the prescanner/preprocessor, but due to its design, this wouldn't be feasible right now.

All the information required to re-format the cooked character stream as valid fixed/free form Fortran source should already be available. Making sure that the debug info based on the "pre-processed" file matches the original file will be much trickier. This could be implemented at a later time though. For now, we can focus on fixing `-E` and the CMake integration.

I agree with Kiran in that intially we could focus on the Makefile generator in CMake. AFAIK, everything that's required is already in place. In order to have a functioning Ninja generator, we will have to implement the reformatting of the "cooked character stream" (or, in other words, fix `-E`).

Please reply here if I missed or misinterpreted anything.

-Andrzej

[1] llvm-project/Overview.md at main · llvm/llvm-project · GitHub
[2] llvm-project/Parsing.md at main · llvm/llvm-project · GitHub

@Tin, given the status of LLVM Flang - support (#22387) · Issues · CMake / CMake · GitLab, I think that LLVMFlang: Add support for LLVM Flang (!6323) · Merge requests · CMake / CMake · GitLab could be re-opened now. Are you still available to work on this?

Thank you,

-Andrzej

Hi Andrzej,

Absolutely. I'll catch up on conversations and reopen the MR soon (within 2 weeks).

Tin

Hi Tin,

Any updates?

Thank you,
-Andrzej

Hi Andrzej,

Thanks for reminding me. From the conversations, it looks to me like I'm good to reopen the MR as is, but with the compiler ID name changed to LLVMFlang instead of FlangLLVM. Is that correct?

Tin

From the conversations, it looks to me like I'm good to reopen the MR as is, but with the compiler ID name changed to LLVMFlang instead of FlangLLVM. Is that correct?

That's how I read too. Thanks for working on this!

-Andrzej