[RFC] Support CMake option to control link type built for Flang runtime libraries

Subprojects Flang
1

The Phabricator review for this RFC is here: :gear: D153426 Support LLVM_BUILD_USER_FORTRAN_LIBS=[shared|static|all] cmake option

For this post, assume that “user Fortran libraries” and “Flang runtime libraries” refer to the same thing, the libraries in question are libFortranRuntime and libFortranDecimal, which are the two libraries linked into user executables that we care to be able to control the link type of. Also note that libFortranDecimal also gets linked into flang-new when building the compiler, this is a primary reason for this patch.

The goal we are trying to achieve through this patch is to have more control over what link types we are building for the Flang runtime libraries, independently of the link types built and linked into the compiler at build time.

There is an important distinction between which library link types are built, and which types are linked into what. We want to separate control over these so that when we are building the compiler we can choose:

  • whether we want LLVM libs to be (built and then) linked as shared or static into flang-new (All LLVM libs, including FortranDecimal)
  • whether to build the Flang runtime libraries (which includes FortranDecimal as well) as static or shared. Note that the usage (actual linking) of these libraries will only occur later by the user of the compiler.

Currently the link type that’s built is primary decided by whether BUILD_SHARED_LIBS is set. CMake seems to default on building the libs as static unless BUILD_SHARED_LIBS is set, in which case it prefers to build them as shared. However, if the call to the llvm_add_library function explicitly specifies a lib type, that type takes precedence over BUILD_SHARED_LIBS, because of this, some libs always get built with a certain link type.

The solution experimented with in the patch linked at the top of this post is using a new CMake variable (currently named LLVM_BUILD_USER_FORTRAN_LIBS=shared|static|all but likely to change to something more correct like FLANG_BUILD_RUNTIME_LIBS instead) that appends additional link types to the call to llvm_add_library, therefore explicitly ensuring that whatever link type is passed by this variable will be built in addition to whatever was determined by BUILD_SHARED_LIBS or any defaults.

Note that we don’t want this new variable to impact how flang-new gets linked. The following case can be troublesome:
Lets say we do not specify BUILD_SHARED_LIBS, we want to statically build and link the LLVM libs into flang-new, however we also specify FLANG_BUILD_RUNTIME_LIBS=shared because we want to dynamically link user executables when using our driver. The problem is that CMake refers to libraries by their name, without the file extension (.a vs .so), so the static and shared libs need to have unique names so they do not cause errors in the CMake code. The code to handle this already exists in AddLLVM.cmake. If the llvm_add_library function is called for a lib named “Foo”, with explicit shared and static libtype arguments (this is what will happen for FortranDecimal and FortranRuntime if FLANG_BUILD_RUNTIME_LIBS=shared is used like in this example) then 2 separate add_library CMake function calls are made, one with the lib name “Foo” and lib type of shared, and one named “Foo_static” with a lib type of static.

This can be problematic because other CMake code is searching for these library targets by name, and if we want to preserve the preference for static linking (since BUILD_SHARED_LIBS was not specified) we now need to make sure that all the usages of libraries that got double built know that they need to use the “_static” variant. Since this new variable only affects a short list of Flang-specific libraries, the patch currently just adds a global property that is set whenever both lib types get built (and therefore “_static” named libs exist) and then this global is checked for at all the usages of FortranRuntime, FortranDecimal (the patch currently also experiments with the idea of a BUILD_STATIC_LIBS variable that is checked, the theory being that if you set BUILD_STATIC_LIBS AND BUILD_SHARED_LIBS thats another case where “_static” libs are created. In retrospect, I don’t think introducing this is necessary, will probably remove from this patch). We also need to ensure that this difference in name is handled somehow so that flang-new will properly link against the “_static” libs if we are explicitly statically linking (ie flang-new --static).

The patch is work-in-progress and I am looking to start discussion about all the information above.

Below is a table covering the behavior I think we want for all the different combinations of the configurations covered in this post. Please let me know if something in this table seems misaligned with other expectations:

BUILD_SHARED_LIBS FLANG_BUILD_RUNTIME_LIBS flang-new --static Expected behavior
TRUE all TRUE All the libs get built as shared. In addition, the Flang runtime libs also get built as static. flang-new will be linked against all the shared libs. The flang-new driver will look for and find the static runtime libs to use at runtime.
TRUE all FALSE Same as above except that since --static is not passed to flang-new, the driver will look for and use the shared libs by default.
TRUE static TRUE Same as the entry 2 rows up. FLANG_BUILD_RUNTIME_LIBS appends the static link type to the list of link types to be built.
TRUE static FALSE Same as entry 2 rows up.
TRUE shared TRUE All the libs get built as shared. FLANG_BUILD_RUNTIME_LIBS does not do anything in addition. The libs are linked into flang-new as shared. When using flang-new --static, the driver throws an error as it is unable to find static versions of the runtime libs.
TRUE shared FALSE Same as previous entry except that the flang-new driver does not throw an error since its looking for and will find the shared libs.
FALSE all TRUE Libs are preferred to build and be linked into the compiler statically by default by CMake if BUILD_SHARED_LIBS is not set. The Flang runtime libs get built additionally as shared. The driver is able to find and use the static libs at runtime.
FALSE all FALSE Same as above except the driver finds and uses the shared libs by default.
FALSE static TRUE All the libs get built statically and linked statically into both the compiler and user executables.
FALSE static FALSE All the libs get built statically and linked statically into the compiler and implicitly into user executables.
FALSE shared TRUE All the libs get built statically and linked statically into the compiler. In addition, the Flang runtime libs get built as shared. The flang-new driver links the static libs into user executables.
FALSE shared FALSE Same as previous entry except the driver finds and uses the shared libs by default.
2

Taking a step back, maybe it’s worth reconsidering the way runtime libraries are built.

Currently, outside of flang, runtime libraries (compiler-rt/libcxx/etc.) are built separately from the compiler. This is facilitated using “LLVM_ENABLE_RUNTIMES”: CMake gets recursively invoked, pointing to the just-built compiler. (Or you can just build each runtime separately.) Support for building the flang runtime separately exists (⚙ D130352 [flang] Allow configuring building the flang runtime standalone) but it’s not the default, and it’s not integrated into the LLVM_ENABLE_RUNTIMES infrastructure.

If we separate the builds this way, the “BUILD_SHARED_LIBS” option for the compiler itself does not affect the runtime libraries; actually, it wouldn’t even exist in the runtime build.

Not saying this should block your work, but recreating existing infrastructure might not be the best use of your time.

1 Like
3

I also believe that your motivation is weak and comes with more complexity in the build system. For C/C++, I consume the runtime libraries installed by the vendor. I am not super keen on being able to link them shared or static.

To make progress, you could first add -static and -shared options to flang.

4

I understand the motivation for this change using the following example:
A linux distribution wants to package LLVM flang and has a policy of in general using BUILD_SHARED_LIBS to build LLVM and only providing those shared libraries (this seems common for linux distributions).
As it stands this would result in the flang runtime libraries only being built as shared libraries (as I understand it).
This would mean that as an end user fortran programmer, my programs built with flang-new provided from the distribution would only be able to dynamically link to the runtime, and so I’d have to ship around that .so everywhere and potentially have issues. Whereas I probably want the option of statically linking the flang runtime to avoid this.

This is my understanding of the motivation, correct me if I’m wrong on any of this!

5

So are you thinking that the focus should instead be shifted on building flang runtimes similar to the other runtime libs (ie integrating it into the LLVM_ENABLE_RUNTIMES infrastructure)? From a preliminary investigation this already seems to present some challenges. It looks like if you build the standalone Runtimes CMake project (llvm-project/runtimes) the cmake code expects all the different runtimes to be found in llvm-project (e.g. llvm-project/libcxxabi), and I’m guessing there is no interest in moving flang’s runtime to be a top-level directory.

There’s a couple questions I have that will help me understand how to approach this (apologize if some of these have obvious answers, I am still fairly new LLVM’s CMake infrastructure):

  • Do we want to build flang’s runtime libs when LLVM_ENABLE_RUNTIMES=all?
  • Do we want to build flang’s runtime libs when something like LLVM_ENABLE_RUNTIMES=flang-rt?
  • Do we want to be able to build flang’s runtime libs as standalone (ie using the llvm-project/runtimes project)
  • Do we want to always build the runtime libs when building the flang project or do we want it to be optional?
  • In all these scenarios, are we treating FortranRuntime and FortranDecimal separately or together (ie, are we dealing with LLVM_ENABLE_RUNTIMES+=;FortranRuntime;FortranDecimal or something combined like LLVM_ENABLE_RUNTIMES+=;flang-rt)
6

A linux distribution wants to package LLVM flang and has a policy of in general using BUILD_SHARED_LIBS to build LLVM and only providing those shared libraries (this seems common for linux distributions). As it stands this would result in the flang runtime libraries only being built as shared libraries (as I understand it).

This seems fairly accurate. For our internal downstream build of flang we are using a script that invokes CMake (currently am building for a linux distribution). This script may pass the BUILD_SHARED_LIBS variable and when it does so, the flang runtime libs (currently) will only get built as shared.

This would mean that as an end user fortran programmer, my programs built with flang-new provided from the distribution would only be able to dynamically link to the runtime, and so I’d have to ship around that .so everywhere and potentially have issues. Whereas I probably want the option of statically linking the flang runtime to avoid this. This is my understanding of the motivation, correct me if I’m wrong on any of this!

Your understanding seems correct, that is one valid case for these changes. Its actually not the exact scenario that sparked this whole investigation but it is similar. What we actually wanted to do was the opposite, we wanted to build and link the libs statically into flang (I believe since its easier to ship the compiler with all its LLVM dependencies linked statically), but have shared runtime libs built for the end user to use (I believe the motivation here was to allow for smaller end user executable sizes). The 2 issues that you encounter when trying this are:

  • if you build flang statically (no BUILD_SHARED_LIBS), the flang runtime libs do not ever get built as shared
  • once you fix this by telling cmake to also build the flang runtime libs as shared (eg through a new cmake variable like we do in the patch), then you must also account for the “_static” name problem, otherwise flang will get linked dynamically to FortranDecimal

So these are the 2 issues that patch addresses, but ultimately we want the ability to configure these choices, not just support the BUILD_SHARED_LIBS=false && FLANG_BUILD_RUNTIME_LIBS=shared scenario. However, based on the discussion in this discourse thread I am seeing that there may be better ways to approach this, with pre-existing the infrastructure used by other runtimes. I have been investigating alternatives (still have a lot of questions) and monitoring this thread for feedback. Based on the other discussion in this thread, what do you think about the current approach vs a new approach using LLVM_ENABLE_RUNTIMES. Is the current approach with FLANG_BUILD_RUNTIME_LIBS a bit misguided?

7

I think the issue with building the flang runtime with LLVM_ENABLE_RUNTIMES is that would require that clang is included in every build of flang. This is something that is required right now for the driver but we want to move away from this requirement.

8

You are already doing a RUNTIMES build for compiler-rt and maybe OpenMP. The Flang runtimes probably can be build with a standard compiler.

9

That’s a decision we’d have to make… but probably.

Yes, we want something like that.

That sort of works automatically given the way the infrastructure works… but yes. Sometimes it’s more convenient to have separate CMake invocations.

I don’t think we need to enforce building the runtime libs, as long as the documentation makes the required steps clear. (Like I mentioned before, it’s useful to be able to build them separately in some cases.)

I think it makes sense to treat “flang-rt” as one thing, unless there’s some reason a user would want one library but not the other.

We could add a flang-runtime top-level directory if that makes things significantly easier? (Wouldn’t necessarily need to contain any code, just the CMake scaffolding.)

Allowing builds with LLVM_ENABLE_RUNTIMES implies allowing standalone builds of the runtime; a standalone build can use any C++ compiler you want. For convenience, we could add some CMake code to allow a single CMake invocation to build both the compiler and the runtime with an arbitrary compiler.

10

That’s a decision we’d have to make… but probably.

So one issue I see with this specifically is that Flang is not even considered one of LLVM_ALL_PROJECTS yet:

From llvm/CMakeLists.txt:

set(LLVM_ALL_PROJECTS "bolt;clang;clang-tools-extra;compiler-rt;cross-project-tests;libc;libclc;lld;lldb;mlir;openmp;polly;pstl")
# The flang project is not yet part of "all" projects (see C++ requirements)
set(LLVM_EXTRA_PROJECTS "flang")
# List of all known projects in the mono repo
set(LLVM_KNOWN_PROJECTS "${LLVM_ALL_PROJECTS};${LLVM_EXTRA_PROJECTS}")
set(LLVM_ENABLE_PROJECTS "" CACHE STRING
    "Semicolon-separated list of projects to build (${LLVM_KNOWN_PROJECTS}), or \"all\".")
# Make sure expansion happens first to not handle "all" in rest of the checks.
if( LLVM_ENABLE_PROJECTS STREQUAL "all" )
  set( LLVM_ENABLE_PROJECTS ${LLVM_ALL_PROJECTS})
endif()

So enabling flang-rt as something thats built when building “all” runtimes seems like overstepping, at least until flang is treated as part of “all” projects

11

I am guessing that that was because when Flang was merged into llvm-project, it used C++17 while the rest of llvm-project was still on C++11. That has since changed, so it should be safe to add Flang to LLVM_ALL_PROJECTS.

12

That has since changed, so it should be safe to add Flang to LLVM_ALL_PROJECTS .

I did some testing and it seems that moving flang to LLVM_ALL_PROJECTS works without issue. I opened up :gear: D154109 Moved flang to LLVM_ALL_PROJECTS to show the change and posted to the flang-compiler slack workspace to start a discussion about the change.

I had a conversation with Peter Klausler who contributed a lot of useful information as well as brought up some concerns with the idea to “integrate into existing LLVM runtime infrastructure”. I will try to summarize the points he brought up here:

  • putting the fortran runtimes into llvm’s runtime model will not be trivial
  • The fact that FortranDecimal is needed for both the runtime and the compiler, as well as the fact that both FortranDecimal and parts of the FortranRuntime library need to be built for gpu devices, make integration into the existing runtime model more complicated.
  • It makes sense to combine FortranDecimal and FortranRuntime into one library for the runtime however this comes with a concern that combining the libraries into a single binary, and then making the compiler depend on that, that would make it really hard to add fortran code to the runtime

Two recommended approaches he brought up were:

  • “maybe what you could do now is just address the need of being able to build both shared and static versions of the two rt libraries independently from the mode of the compiler’s build. consider unconditionally building the libraries in both modes, that might do it.”
  • “or alternatively: establish a new library for the compiler that has a distinct name from the current decimal conversion library, but is built from the same sources, and make that new library part of the compiler build.”

I am relaying these suggestions here in case anyone has further opinions. The sentiment still seems fairly split when it comes to which approach we should take with this so I would like to keep the conversation going in hopes in reaching a more decisive conclusion.

13

I think the notion of treating the compiler’s version of FortranDecimal as a separate library from the Fortran runtime makes sense. The build flags you want to use for a library linked into the compiler are probably different from the flags you want for a runtime library. For example, the preferred settings for optimization level, whether LTO is enabled, the chosen symbol visibility. It also makes the build easier to understand because there’s a clear separation between “host” and “target” libraries.

This is sort of similar to the issues we have with the C++ demangler, which we also end up building twice. One version goes into LLVM utilities like llvm-cxxfilt, the other goes into libc++. (We actually have two copies of the source code, but the reasons for that probably aren’t relevant here.)

I guess splitting the build like that could be treated as an incremental step towards supporting an LLVM_ENABLE_RUNTIMES build.

Not sure I understand how GPU runtimes are different from CPU runtimes.

1 Like
14

I have been experimenting with the ideas discussed last week and currently have a prototype for the following, looking for feedback before I carry on with finalizing the code:

  1. FortranDecimal gets built as 2 separate library targets, FortranDecimal (for the compiler) and FortranDecimalRT (for the runtime). This change seemed trivial but I still need to fully test it and make sure things work as expected.
  2. I built upon 1 by also taking a shot at the top-level flang-rt directory idea that was discussed earlier. The idea with this being, in order to more fluidly accommodate the existing llvm runtimes infrastructure, CMake scaffolding to create a FlangRT library exists in llvm-project/flang-rt. This accomplished 2 goals:
    a) FlangRT combines FortranDecimalRT and FortranRuntime into a single library for simplicity
    b) a top-level directory for the flang runtime now exists similarly to all the other runtimes in LLVM, and can be built with LLVM_ENABLE_RUNTIMES without any changes to that infrastructure.
    FlangRT just combines the 2 libraries without adding any additional sources.

I want to make sure the community is really in favor of 2 before I iron out any remaining issues and test it thoroughly, since its fairly impactful.

15

Update on what’s been happening with the Flang runtime investigation

It seems that the course of action that the community is leaning towards is:

  1. Split FortranDecimal into 2 separate library targets. Same sources but 1 library is used by the compiler and the other one at runtime.
    Reasoning:
    • decouple compiler and runtime so that for example, different build flags (such as link type) can be specified for compiler builds and runtime builds
  2. Define FlangRT target: the final runtime library for flang, built using the existing LLVM runtime infrastructure.
    Reasoning/Goals:
    • Combines FortranRuntime and FortranDecimalRT into single library for end user
    • Integrates flang’s runtime into the existing runtime infrastructure
    • Enables standalone builds of the flang runtime (Build using cmake target llvm-project/flang-rt or llvm-project/runtimes with LLVM_ENABLE_RUNTIMES=flang-rt)
    • Building flang as usual will also build flang-rt as an enabled runtime by default (when building cmake target llvm-project/flang or llvm-project/llvm with LLVM_ENABLE_PROJECTS=flang

Approaches for FlangRT:

  • Original approach: Wrapper library - As previously discussed in this thread and on slack, create a new library with cmake, with a dummy source file (seems needed in order to trick cmake into creating a library), and then target_link_libraries FortranRuntime and FortranDecimalRT into this new FlangRT library.
  • Current new approach: Object Libraries (add_library — CMake 3.27.0-rc4 Documentation) - By building FortranRuntime, FortranDecimalRT as object libraries and adding the objects as sources to FlangRT, this seems like a more direct and clean approach to building FlangRT than by target_link_libraries FortranDecimalRT, FortranRuntime. This also avoids the need for a dummy source file, and also still keeps the sources of FortranDecimalRT and FortranRuntime decoupled (as opposed to something like literally adding all sources from Decimal and Runtime directly into FlangRT).

Regardless of approach, the idea is that FlangRT is looked for by the driver instead of FortranRuntime and FortranDecimalRT

Currently, I am working through debugging some linker bugs when trying to use FlangRT built by the current approach. (However, the infrastructure and the bulk of the code is finished and FlangRT builds). I’m not sure yet if the issue I’m facing is because of problems with the approach or if its just an error in my code I haven’t found yet.

Using the object libraries approach hasn’t been properly discussed with the community yet, so I want to bring this up now to make sure its even viable. I am still relatively new to cmake and build systems in general, so if there are any concerns with using object libraries please let me know.

I would like to post a patch of the WIP soon, but since at the moment it still has some bug(s) I am hesitant to post a patch yet, since I wouldn’t want anyone to assume the code is functioning and don’t know exactly what the LLVM policy is regarding posting phabricator reviews for patches that are known to be not-functional yet.

16

“Same sources” means that there’s just one source directory, not two directories with what are currently identical source files, yes?

18

Yes 1 source directory, literally just 2 separate “add_library” calls executed by cmake

1 Like
19

I don’t really see anything wrong with this? It’s a little ugly, but the resulting CMake is simple and readable.

If object libraries help, feel free to use them… but I don’t see how they do; each built object file should end up in exactly one library.

It’s fine to post something without intent to merge it as long as it’s relevant for LLVM development.

20

:gear: D154869 [Flang] [FlangRT] Implement FlangRT library as solution to Flang’s runtime LLVM integration

Ok then, here is the patch in its current form. I’ve left in quite a few comments and TODOs

21

This patch in its current state, is now a working implementation of flang-rt.

  • flang-rt is built as an enabled runtime when building cmake llvm LLVM_ENABLE_PROJECTS=flang
  • flang-rt can be built as a standalone runtime when building cmake runtimes LLVM_ENABLE_RUNTIMES=flang-rt
  • FLANG_RT_ENABLED_SHARED=On and FLANG_RT_ENABLE_STATIC=On can be used to determine flang-rt’s link type, both can be present, in which case, just like other llvm libraries, flang-rt gets 2 builds, the shared is called libflang-rt.so, and the static is libflang-rt_static.a (in CMake code the target names must differ)
  • the flang driver needs to find flang-rt for linking into user executables, remember to set LD_LIBRARY_PATH if building flang-rt as shared (see [flang][driver] Add support for generating executables · llvm/llvm-project@97a32d3 (github.com)). FortranDecimalRT and FortranRuntime are not direct dependencies of flang-new
    Speaking of FortranDecimalRT, this patch introduces this target as a duplicate of FortranDecimal (exact same source files) but it is only built for flang-rt, this decouples the runtime and compiler.
  • Since Fortran_main is necessary to implement the main entry point into Fortran’s PROGRAM in Flang, and needs to be built statically, Fortran_main is still separate from flang-rt.
1 Like