RFC: Caffeine parallel runtime library

I believe this is my first awareness of this Discourse subproject so my apologies for not posting here sooner. @kiranchandramohan suggested in the flang-compiler Slack workspace that I share plans here for Caffeine, which the Berkeley Lab Flang team proposes as a parallel runtime library for LLVM Flang and potentially for other compilers. (We’ve had some discussions with the LFortran compiler developers also.)

We presented an overview of the Berkeley Lab Flang project, including Caffeine, in a research poster last week at the SC22 conference. Our project is a three-part effort using test-driven development to accelerate compiler support for parallel Fortran 2018 features.

  1. Kate Rasmussen leads our frontend work, which is focused on writing unit tests for the compile-time semantics of parallel features. Kate contributes frontend fixes for issues exposed by the tests. A related GitHub project board shows 36 issues closed, 4 under development, and 4 “To Do”.
  2. I lead the parallel runtime work with a lot of great guidance from Dan Bonachea. For more details, please see the paper that I presented at the LLVM for HPC Workshop at SC22.
  3. Hugh Kadhem leads our work on lowering parallel features. The lowering work is just getting started.

We are open to whether Caffeine becomes an external dependency or gets merged into LLVM Flang. Either way, we hope that the path forward preserves the option to support multiple compilers and to provide multiple alternative backends. One alternative backend that I’d like to see at some point is OpenCoarrays, which would facilitate native Windows support and would provide near-complete support for Fortran 2018 parallel features. Allowing alternative backends also opens up the possibility that hardware vendors might provide alternative runtimes optimized for their hardware.



Thanks for sharing your plans for the Coarray work. I think it will be useful to get an agreement with the core developers, code owner and the rest of the community.

The frontend work is probably agnostic of the runtime being used, so I guess making progress there is non-controversial. When it comes to lowering, it is encouraged to submit a design document as detailed in llvm-project/DesignGuideline.md at main · llvm/llvm-project · GitHub. It might be interesting to know whether what you are planning has a new MLIR component (like a co-array dialect) or whether you will be lowering to FIR + runtime calls to the caffeine runtime.

I believe Caffeine is mostly written in Fortran, so that it can be used by multiple compilers without having to worry about compatibility issues regarding module files, array descriptors etc. Currently, the Flang runtime is written in C++ (independent of the C++ library). Would there be any issues in using a part of the runtime in Fortran? The lowering code for Co-arrays will probably reside in https://github.com/llvm/llvm-project/blob/main/flang/lib/Lower/Coarray.cpp. I don’t know whether this means that progress can be made in lowering Coarrays without affecting the rest of the parts and whether we can add support for multiple co-array runtimes. I am assuming the runtimes are not standardised and will have different interfaces.

Contributing the runtime to LLVM is probably better for integrating it with the Flang compiler. But I don’t know whether there will be more contributors to the runtime if it is kept outside the llvm-project. Since LFortran also uses LLVM, they might be OK to have the runtime sit in LLVM. I don’t know whether gfortran would have issues in using this.

It will be interesting to also see whether existing Fortran vendors with a Coarray implementation have plans for contributing their runtime or alternatively participating in this work.

Thanks for sharing your plan.

To add to what @kiranchandramohan said earlier, I think it will be good to have an intermediate layer when you lower coarray. It can be a dedicated dialect for coarrays or at least specific FIR operation so it separates lowering from Caffeine itself. I think this is even more important if caffeine is not integrated in LLVM/Flang but a dependency.

I would like to see the f18 lowering/runtime API for remote load/store, synchronization, teams, liveness, &c. be organized around Fortran concepts rather than specific runtime models and implementations, so that connecting to proprietary vendor interconnects, open distributed runtimes, single-image execution, & SMP shared memory execution can all take place by means of an API internal to the runtime (where other current Fortran-centric lowering/runtime APIs will need to interact with coarrays for allocatable coarray allocation, deallocation, &c.) and allow for single compilations of executables and libraries. So what’s important now is having a minimized Fortran lowering/runtime API and its concomitant extensions to the single f18 descriptor and its addendum.

This sounds like xkcd. The existing standards are bad. Let’s do the next one.

The MLIR solution is: Let’s design a Coarray dialect. Lower all Coarray constructs to the dialect. Maybe there are even optimisations opportunities. Then offer lowering passes from the dialect to the well-known runtime APIs. Vendors can add there own lowering passes to their secret runtime APIs.

@kiranchandramohan thanks for referring me to the Design Guideline. The LLVM for HPC Workshop paper has much of the content recommended in the guideline so hopefully we can borrow a lot content from the paper to get going with a design document. The main thing missing is any discussion of lowering because I have zero expertise in that arena, but I specifically brought in our team member Hugh Kadhem because of his background in that area from working on the IBM compiler project when they were retooling to sue the LLVM backend. I hope we can respond in more detail after the holiday. I agree with others that the API should be defined independently of the runtime. I think the ability to support multiple runtime alternatives would be a really nice feature for LLVM flang.

We Fujitsu has a proprietary Coarray implementation for our platforms (Fugaku and commercial machines). We plan to switch our proprietary Fortran compiler to Flang/LLVM in the future.

Regarding @kiranchandramohan’s question, we don’t have a plan to contribute our runtime or use our runtime with Flang. It is tightly coupled to our compiler and the design is a bit old. We have an interest in the Caffeine work but don’t have enough power to participate because we are focusing our power to middle-end/backend optimization. We may be able to provide Fortran 2008 coarray test programs and design documents if the community needs them (though we have to translate Japanese to English).

1 Like