Building Flang with offload support

Subprojects Flang
1

I have had an issue for some time which have prevented me from enabling OpenMP offloading in my main application (this is/was blocking issue 1 of 2). I had some big hopes that this would now be fixed when mod-files are built per target (thanks @Meinersbur - I have been looking forward to this for a long time!), but I am still struggling.

I now configure and build LLVM and Flang like this:

cmake -GNinja \
    -DCMAKE_BUILD_TYPE="Release" \
    -DCMAKE_INSTALL_PREFIX="/opt/llvm/llvm-project/install" \
    -DCMAKE_CUDA_COMPILER=${CUDACXX} \
    -DCMAKE_CUDA_HOST_COMPILER="clang++-20" \
    -DCMAKE_CUDA_ARCHITECTURES="89;120" \
    -DLLVM_ENABLE_PROJECTS="clang;lld;flang;lldb" \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;openmp;offload;flang-rt" \
    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \
    -DLLVM_ENABLE_RTTI="ON" \
    -DLLVM_RUNTIME_TARGETS='default;nvptx64-nvidia-cuda' \
    -DOPENMP_ENABLE_LIBOMPTARGET="ON" \
    -DFLANG_RT_EXPERIMENTAL_OFFLOAD_SUPPORT="OpenMP" \
    -DFLANG_RT_DEVICE_ARCHITECTURES="sm_89;sm_120" \
    -DFLANG_RT_ENABLE_STATIC=ON \
    ../llvm 2>&1 | tee cmake.log
nice ninja -j 12 install

Building the reproducer from issue 146876 with offloading enabled:

program real10
  use ieee_arithmetic
  implicit none
end program

now gives a series of linker errors:

$ flang --version
flang version 22.0.0git (https://github.com/llvm/llvm-project.git 7675fc79c802cf7f6a95660f6ee59bf6cb62102f)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/llvm/llvm-project/install/bin

$ flang -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda real10.F90
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_away' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_denorm' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_divide_by_zero' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_down' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_inexact' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_invalid' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_nearest' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_negative_denormal' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_negative_inf' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_negative_normal' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_negative_subnormal' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_negative_zero' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_other' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_other_value' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_overflow' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_positive_denormal' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_positive_inf' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_positive_normal' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_positive_subnormal' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_positive_zero' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_quiet_nan' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QMieee_arithmeticECieee_signaling_nan' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_to_zero' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_underflow' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
nvlink error   : Undefined reference to '_QM__fortran_builtinsEC__builtin_ieee_up' in '/tmp/real10-054f3c-nvptx64-nvidia-cuda-sm_89-f50675-0bef9e.cubin'
/opt/llvm/llvm-project/install/bin/clang-nvlink-wrapper: error: 'nvlink' failed
clang: error: nvlink command failed with exit code 1 (use -v to see invocation)
/opt/llvm/llvm-project/install/bin/clang-linker-wrapper: error: 'clang' failed
flang-22: error: linker command failed with exit code 1 (use -v to see invocation)

I suspect that the problem is that no runtime library for the target/device architecture is being built. I have tried to set -DRUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES="compiler-rt;flang-rt;openmp", but this breaks every small example using offloading by introducing additional linker errors, i.e. every small OpenMP example suddenly do not link with the error:

$ flang -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda loop-01-index.F90 
/opt/llvm/llvm-project/install/bin/clang-nvlink-wrapper: error: unable to find library -lflang_rt.runtime
clang: error: nvlink command failed with exit code 1 (use -v to see invocation)
/opt/llvm/llvm-project/install/bin/clang-linker-wrapper: error: 'clang' failed
flang-22: error: linker command failed with exit code 1 (use -v to see invocation)

So this option really only introduce additional problems. Is there an updated documentation on how to build LLVM+Flang with as much offloading support as possible? Do I do something wrong when building Flang? Or is what I try to do still just not supported/working?

Thanks in advance for any hints.

2

I played around with this a bit and found that at least for that specific reproducer, the undefined symbols go away with optimization level O1 or higher. With O0 I get the same linker error.

Looking at the generated LLVM code I see that there is code like this generated for those unused variables when the file is compiled with O0:

source_filename = "FIRModule"
target datalayout = "e-p6:32:32-i64:64-i128:128-i256:256-v16:16-v32:32-n16:32:64"
target triple = "nvptx64-nvidia-cuda"

%_QM__fortran_builtinsT__builtin_ieee_round_type = type { i8 }

@_QM__fortran_builtinsEC__builtin_ieee_away = external constant %_QM__fortran_builtinsT__builtin_ieee_round_type

There seems to be a difference in how those unused variables are handled by the NVPTX backend compared to AMDGPU and x86.

Compiling that code with llc generates the following PTX code which results in an object where the symbol is undefined if assembled with ptxas:

.version 7.8
.target sm_89
.address_size 64

.extern .global .align 8 .b8 _QM__fortran_builtinsEC__builtin_ieee_away[1];

Doing the same exercise with x86 does not create the symbol.

source_filename = "FIRModule"
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"

%_QM__fortran_builtinsT__builtin_ieee_round_type = type { i8 }
@_QM__fortran_builtinsEC__builtin_ieee_away = external constant %_QM__fortran_builtinsT__builtin_ieee_round_type

Compiling the above with llc produces:

	.file	"FIRModule"
	.section	".note.GNU-stack","",@progbits
1 Like
3

I haven’t done a large amount of testing on this, and I think there are still places actively in development here. You’re looking for a GPU version of the Flang runtime, there’s the historical one which routes this through CUDA and my version which routes it through my custom GPU toolchain, that seems to be the one you’re trying to use. Using this requires a far more complicated stack, as flang-rt depends on C++ headers. That means at a minimum you need to build those for the GPU and then tell flang-rt you intend to use the headers you just generated instead of ones found in the system. I haven’t tested this in awhile, but I’d assume it’d be something like this.

    -DRUNTIMES_nvptx64-nvidia-cuda_LLVM_ENABLE_RUNTIMES=compiler-rt;libc;libcxx;libcxxabi;flang-rt;openmp \
    -DRUNTIMES_amdgcn-amd-amdhsa_LLVM_ENABLE_RUNTIMES=compiler-rt;libc;libcxx;libcxxabi;flang-rt;openmp \
    -DRUNTIMES_nvptx64-nvidia-cuda_FLANG_RT_LIBC_PROVIDER=llvm \
    -DRUNTIMES_nvptx64-nvidia-cuda_FLANG_RT_LIBCXX_PROVIDER=llvm \
    -DRUNTIMES_amdgcn-amd-amdhsa_FLANG_RT_LIBC_PROVIDER=llvm \
    -DRUNTIMES_amdgcn-amd-amdhsa_FLANG_RT_LIBCXX_PROVIDER=llvm \
    -DRUNTIMES_nvptx64-nvidia-cuda_CACHE_FILES=${LLVM_SRC}/../libcxx/cmake/caches/NVPTX.cmake \
    -DRUNTIMES_amdgcn-amd-amdhsa_CACHE_FILES=${LLVM_SRC}/../libcxx/cmake/caches/AMDGPU.cmake

Quite the mouthful, in the future we should probably get a testing configuration for this up and provide a cache file. The ROCm compiler just recently switched to using the latter method, so I assume it works for AMDGPU at least.

4

Thank you all for the comments and feedback. Since the change to move builtin .mod generation into runtimes was reverted a 2nd time, I have put this on hold until the relevant PR can be merged a third (and hopefully last) time.

When the PR gets merged I’ll once again pick up the last main branch and try to see how it works.