RFC: Is `flang-new` ready to replace `f18`?

Hello,

As the new Flang driver has recently reached feature parity with the "throwaway" driver, I wanted to ask whether we are ready to remove `f18`?

To better inform this discussion, below you will find an up-to-date status update. I wanted to avoid sending separate e-mails - I hope that this fine!

*SUPPORTED OPTIONS*
A summary of the supported options is available in [1]. I shared that spreadsheet in November last year [2]. I've been updating it while new features were being added. Most notable changes:
* `-fdebug-no-semantics` is replaced with 2 options: `-fdebug-unparse-no-sema` [3] and `-fdebug-dump-parse-tree-no-sema` [4]
* `-fconvert=swap/-byteswapio` is not supported in the "throwaway" driver [5] and hence has not been added in `flang-new`
* `-ffree-line-length=<arg>` is currently not supported by the frontend and hence has not been added in `flang-new`
* `-futf-8`/`-flatin` are replaced with `-finput-charset=<val>` (the latter is the GCC/Clang spelling that we didn't know about when compiling [1])
As discussed previously [2], `flang-new` uses GCC style spellings.

*TESTING*
Apart from adding tests specifically for the new driver, all existing Flang tests have been ported to use `flang-new` when `FLANG_BUILD_NEW_DRIVER` is set. In order to facilitate this, 2 new LIT variables have been added. Their meaning depend on whether `FLANG_BUILD_NEW_DRIVER` was set or not (unset by default):
* `%flang` expands to either `flang-new` or `f18` and represents compiler driver
* `%flang_fc1` expands to either `flang-new -fc1` or `f18` and represents frontend driver
This is similar to `%clang` and `%clang_cc1` in Clang and allows a clear separation between the drivers when testing. Please use `%flang_fc1` or `%flang` in your tests from now on. `%f18` will be removed shortly [6]. In terms of upstream testing, all but 1 of our Buildbot workers already set `FLANG_BUILD_NEW_DRIVER` to `On` (i.e. test with the new rather than the old driver).

*NEW VS OLD DRIVER - COMPARISON*
In terms of _driving the frontend_, `flang-new -fc1` and `f18` are fully compatible. Otherwise we wouldn't be able to run the frontend tests with the new driver.

Comparing `flang-new` and `f18` in terms of _driving the whole compiler_ (frontend/codegen/asm/linking) is tricky. The former does not support code-generation yet (it's not available in llvm-project/flang yet). The latter calls a separate Fortran compiler to do the code-generation, so that's more like an experimental feature. This won't be supported in `flang-new`, as discussed in [2].

Lastly, the new driver introduces a dependency on Clang. The steps to remove this dependency were outlined in [7], but we've not been able to make any progress here.

*ADDING NEW OPTIONS*
All options for the new Flang driver are defined in Options.td in Clang [8]. I will be documenting the steps required to add new options in the coming weeks. In the meantime, please feel free to ask me directly (Slack/e-mail) if you have any questions. You can also use one of the existing options for reference.

*NEXT STEPS*
Maintaining multiple drivers is not ideal and we would like to replace `f18` with `flang-new` sooner rather than later. We are keen do it now, despite the differences between `flang-new` and `f18` highlighted above. The new driver covers all use cases that are relevant for us, but we appreciate that this might not be the case for other community members.

Taking the above into account, do you think that `flang-new` is ready to replace `f18`? If not, what's the blocking issue for you? Is it:
* the ability to call an external compiler?
* the dependency on Clang?
* something else?
Please let me know if there's anything that's unclear!

Thank you for reading and for your feedback!

-Andrzej

[1] LLVM/flang new driver options - Google Sheets
[2] [flang-dev] RFC: new Flang driver options
[3] llvm-project/omp-allocate-unparse.f90 at main · llvm/llvm-project · GitHub
[4] llvm-project/dump-parse-tree-no-sema.f90 at main · llvm/llvm-project · GitHub
[5] llvm-project/f18.cpp at b6db6f5530d2df8bc3a1acf6d68b2bfa2bd053b4 · llvm/llvm-project · GitHub
[6] ⚙ D101281 [flang] Remove `%f18` from LIT configuration files
[7] [cfe-dev] RFC: refactoring libclangDriver/libclangFrontend to share with Flang
[8] llvm-project/Options.td at main · llvm/llvm-project · GitHub

Hi,

This is just a quick summary of the discussion that we had on this RFC on Slack and during our most recent community call.

* Peter Klausler and Steve Scalpone do use `flang` (the bash wrapper script for `f18`) to call external compilers. They suggest that we find a way to replicate this with `flang-new` (the new driver) before `f18` can be removed.
* Peter Klausler asked about the dependency on Clang and the ability to build out-of-tree. Just to clarify: the new driver can be built out-of-tree and we have a buildbot to make sure that this works fine: Buildbot. The dependency on Clang will take a while to be removed. There's been no active development in this direction since [1]
* Johannes Doerfert suggested that the dependency on Clang shouldn't be a blocker. Flang will depend on Clang regardless of the new driver (through the OpenMP runtime). Also, given previous design decisions, it feels that compiler build times are not the highest priority for Flang.

Please correct me if I misunderstood anything or missed any comments.

Thank you for your feedback!
-Andrzej

[1] [cfe-dev] RFC: refactoring clangDriver - diagnostics classes

Hello all,

Here's a quick update on what's happened since my last e-mail.

In the previous Flang community technical call we discussed enabling the new Flang driver by default. This change has now been merged upstream: ⚙ D101842 [flang][cmake] Enable the new driver by default. Please remember to set `FLANG_BUILD_NEW_DRIVER` to `Off` if you don't want to build the new driver (and to avoid the dependency on Clang).

Our next step is to find a way to use flang-new, the new driver, to `unparse` source files before calling an external Fortran compiler for code-generation. That's one feature that's available in `f18` and which we won't replicate in `flang-new`. Instead, we'll try to implement this in a wrapper script. Once this is ready, we'll suggest that `f18` is removed.

Please let me know if you have any questions!

Thank you,
-Andrzej

Hello,

The "next step" that I mentioned in my previous e-mail is implemented here: ⚙ D103177 [flang][driver] Extend the `flang` bash script to act as a driver. Please review - your feedback is much appreciated!

To keep things simple, I suggest that we continue this discussion on Phabricator.

Thank you,
Andrzej