Compiling Flang for GPU offloading (again...)

Subprojects Flang
1

I had an older post regarding another build issue, and was in doubt if I should bring up the old post or make a new one. I decided to make a new one, since the problem seems to be something else.

We are trying to follow the Flang development, and regularly build Flang from git sources. In the last weeks we have been unsuccessful building Flang with offloading enabled.

The way we try to build LLVM + Flang now, is:

cmake -GNinja \
    -DCMAKE_BUILD_TYPE="Release" \
    -DCMAKE_INSTALL_PREFIX="/opt/llvm/llvm-project/install" \
    -DCMAKE_CUDA_COMPILER=${CUDACXX} \
    -DCMAKE_CUDA_ARCHITECTURES="89;120" \
    -DLLVM_ENABLE_PROJECTS="clang;lld;flang;lldb" \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;openmp;offload;flang-rt" \
    -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" \
    -DLLVM_ENABLE_RTTI="ON" \
    -DOPENMP_ENABLE_LIBOMPTARGET="ON" \
    -DFLANG_RT_EXPERIMENTAL_OFFLOAD_SUPPORT="OpenMP" \
    -DFLANG_RT_DEVICE_ARCHITECTURES="sm_89;sm_120" \
    -DFLANG_RT_ENABLE_STATIC=ON \
    ../llvm

This fails with a linking error:

[219/225] Linking CXX shared library /opt/llvm/llvm-project/build/lib/libomptarget.so.23.0git
FAILED: /opt/llvm/llvm-project/build/lib/libomptarget.so.23.0git 
: && /opt/llvm/llvm-project/build/bin/clang++ --target=x86_64-unknown-linux-gnu -fPIC -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete   -Wl,--gc-sections  -fopenmp -fvisibility=hidden -fopenmp-cuda-mode --offload-arch=sm_89,sm_120 -foffload-lto -Xlinker --dependency-file=offload/libomptarget/CMakeFiles/omptarget.dir/link.d -shared -Wl,-soname,libomptarget.so.23.0git -o /opt/llvm/llvm-project/build/lib/libomptarget.so.23.0git offload/libomptarget/CMakeFiles/omptarget.dir/device.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/interface.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/omptarget.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/OffloadRTL.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/LegacyAPI.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/PluginManager.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/DeviceImage.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/API.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/Mapping.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/InteropAPI.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/OpenMP/OMPT/Callback.cpp.o offload/libomptarget/CMakeFiles/omptarget.dir/KernelLanguage/API.cpp.o  -Wl,-rpath,"\$ORIGIN:/opt/llvm/llvm-project/build/runtimes/runtimes-bins/offload/libomptarget/..:/opt/llvm/llvm-project/build/runtimes/runtimes-bins/openmp/runtime/src:"  openmp/runtime/src/libomp.so  /opt/llvm/llvm-project/build/lib/libLLVMFrontendOpenMP.a  /opt/llvm/llvm-project/build/lib/libLLVMSupport.a  /opt/llvm/llvm-project/build/lib/libLLVMObject.a  -Wl,--version-script=/opt/llvm/llvm-project/offload/libomptarget/exports  /opt/llvm/llvm-project/build/lib/libomptarget.rtl.amdgpu.a  /opt/llvm/llvm-project/build/lib/libomptarget.rtl.cuda.a  /opt/llvm/llvm-project/build/lib/libomptarget.rtl.host.a  -lm  /opt/llvm/llvm-project/build/lib/libLLVMExtensions.a  /opt/llvm/llvm-project/build/lib/libLLVMPasses.a  /opt/llvm/llvm-project/build/lib/libLLVMCFGuard.a  /opt/llvm/llvm-project/build/lib/libLLVMGlobalISel.a  /opt/llvm/llvm-project/build/lib/libLLVMCoroutines.a  /opt/llvm/llvm-project/build/lib/libLLVMHipStdPar.a  /opt/llvm/llvm-project/build/lib/libLLVMIRPrinter.a  /opt/llvm/llvm-project/build/lib/libLLVMNVPTXCodeGen.a  /opt/llvm/llvm-project/build/lib/libLLVMipo.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendOpenMP.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendOffloading.a  /opt/llvm/llvm-project/build/lib/libLLVMObjectYAML.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendAtomic.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendDirective.a  /opt/llvm/llvm-project/build/lib/libLLVMInstrumentation.a  /opt/llvm/llvm-project/build/lib/libLLVMLinker.a  /opt/llvm/llvm-project/build/lib/libLLVMVectorize.a  /opt/llvm/llvm-project/build/lib/libLLVMSandboxIR.a  /opt/llvm/llvm-project/build/lib/libLLVMSelectionDAG.a  /opt/llvm/llvm-project/build/lib/libLLVMAsmPrinter.a  /opt/llvm/llvm-project/build/lib/libLLVMCodeGen.a  /opt/llvm/llvm-project/build/lib/libLLVMScalarOpts.a  /opt/llvm/llvm-project/build/lib/libLLVMAggressiveInstCombine.a  /opt/llvm/llvm-project/build/lib/libLLVMInstCombine.a  /opt/llvm/llvm-project/build/lib/libLLVMCGData.a  /opt/llvm/llvm-project/build/lib/libLLVMBitWriter.a  /opt/llvm/llvm-project/build/lib/libLLVMObjCARCOpts.a  /opt/llvm/llvm-project/build/lib/libLLVMTransformUtils.a  /opt/llvm/llvm-project/build/lib/libLLVMTarget.a  /opt/llvm/llvm-project/build/lib/libLLVMAnalysis.a  /opt/llvm/llvm-project/build/lib/libLLVMProfileData.a  /opt/llvm/llvm-project/build/lib/libLLVMSymbolize.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoGSYM.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoPDB.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoMSF.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoBTF.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendHLSL.a  /opt/llvm/llvm-project/build/lib/libLLVMCodeGenTypes.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoDWARF.a  /opt/llvm/llvm-project/build/lib/libLLVMObject.a  /opt/llvm/llvm-project/build/lib/libLLVMIRReader.a  /opt/llvm/llvm-project/build/lib/libLLVMBitReader.a  /opt/llvm/llvm-project/build/lib/libLLVMAsmParser.a  /opt/llvm/llvm-project/build/lib/libLLVMCore.a  /opt/llvm/llvm-project/build/lib/libLLVMRemarks.a  /opt/llvm/llvm-project/build/lib/libLLVMBitstreamReader.a  /opt/llvm/llvm-project/build/lib/libLLVMMCParser.a  /opt/llvm/llvm-project/build/lib/libLLVMTextAPI.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoCodeView.a  /opt/llvm/llvm-project/build/lib/libLLVMNVPTXDesc.a  /opt/llvm/llvm-project/build/lib/libLLVMNVPTXInfo.a  /opt/llvm/llvm-project/build/lib/libLLVMMC.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoDWARFLowLevel.a  /opt/llvm/llvm-project/build/lib/libLLVMBinaryFormat.a  /opt/llvm/llvm-project/build/lib/libLLVMTargetParser.a  /opt/llvm/llvm-project/build/lib/libLLVMSupport.a  /opt/llvm/llvm-project/build/lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libzstd.so  /opt/llvm/llvm-project/build/lib/clang/23/lib/x86_64-unknown-linux-gnu/libflang_rt.runtime.a  /usr/lib/x86_64-linux-gnu/libffi.a && :
/opt/llvm/llvm-project/build/bin/clang-linker-wrapper: error: unable to find library -lomp
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
[220/225] Linking CXX shared library /opt/llvm/llvm-project/build/lib/libLLVMOffload.so.23.0git
FAILED: /opt/llvm/llvm-project/build/lib/libLLVMOffload.so.23.0git 
: && /opt/llvm/llvm-project/build/bin/clang++ --target=x86_64-unknown-linux-gnu -fPIC -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -O3 -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete   -Wl,-rpath-link,/opt/llvm/llvm-project/build/./lib  -Wl,--gc-sections  -fopenmp -fvisibility=hidden -fopenmp-cuda-mode --offload-arch=sm_89,sm_120 -foffload-lto -Xlinker --dependency-file=offload/liboffload/CMakeFiles/LLVMOffload.dir/link.d -shared -Wl,-soname,libLLVMOffload.so.23.0git -o /opt/llvm/llvm-project/build/lib/libLLVMOffload.so.23.0git offload/liboffload/CMakeFiles/LLVMOffload.dir/src/OffloadLib.cpp.o offload/liboffload/CMakeFiles/LLVMOffload.dir/src/OffloadImpl.cpp.o  -Wl,-rpath,"\$ORIGIN:/opt/llvm/llvm-project/build/runtimes/runtimes-bins/offload/liboffload/..:"  /opt/llvm/llvm-project/build/lib/libLLVMFrontendOpenMP.a  /opt/llvm/llvm-project/build/lib/libLLVMSupport.a  /opt/llvm/llvm-project/build/lib/libomptarget.rtl.amdgpu.a  /opt/llvm/llvm-project/build/lib/libomptarget.rtl.cuda.a  /opt/llvm/llvm-project/build/lib/libomptarget.rtl.host.a  -Wl,--version-script=/opt/llvm/llvm-project/offload/liboffload/exports  /opt/llvm/llvm-project/build/lib/libLLVMExtensions.a  /opt/llvm/llvm-project/build/lib/libLLVMPasses.a  /opt/llvm/llvm-project/build/lib/libLLVMCFGuard.a  /opt/llvm/llvm-project/build/lib/libLLVMGlobalISel.a  /opt/llvm/llvm-project/build/lib/libLLVMCoroutines.a  /opt/llvm/llvm-project/build/lib/libLLVMHipStdPar.a  /opt/llvm/llvm-project/build/lib/libLLVMIRPrinter.a  /opt/llvm/llvm-project/build/lib/libLLVMNVPTXCodeGen.a  /opt/llvm/llvm-project/build/lib/libLLVMipo.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendOpenMP.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendOffloading.a  /opt/llvm/llvm-project/build/lib/libLLVMObjectYAML.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendAtomic.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendDirective.a  /opt/llvm/llvm-project/build/lib/libLLVMInstrumentation.a  /opt/llvm/llvm-project/build/lib/libLLVMLinker.a  /opt/llvm/llvm-project/build/lib/libLLVMVectorize.a  /opt/llvm/llvm-project/build/lib/libLLVMSandboxIR.a  /opt/llvm/llvm-project/build/lib/libLLVMSelectionDAG.a  /opt/llvm/llvm-project/build/lib/libLLVMAsmPrinter.a  /opt/llvm/llvm-project/build/lib/libLLVMCodeGen.a  /opt/llvm/llvm-project/build/lib/libLLVMScalarOpts.a  /opt/llvm/llvm-project/build/lib/libLLVMAggressiveInstCombine.a  /opt/llvm/llvm-project/build/lib/libLLVMInstCombine.a  /opt/llvm/llvm-project/build/lib/libLLVMCGData.a  /opt/llvm/llvm-project/build/lib/libLLVMBitWriter.a  /opt/llvm/llvm-project/build/lib/libLLVMObjCARCOpts.a  /opt/llvm/llvm-project/build/lib/libLLVMTransformUtils.a  /opt/llvm/llvm-project/build/lib/libLLVMTarget.a  /opt/llvm/llvm-project/build/lib/libLLVMAnalysis.a  /opt/llvm/llvm-project/build/lib/libLLVMProfileData.a  /opt/llvm/llvm-project/build/lib/libLLVMSymbolize.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoGSYM.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoPDB.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoMSF.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoBTF.a  /opt/llvm/llvm-project/build/lib/libLLVMFrontendHLSL.a  /opt/llvm/llvm-project/build/lib/libLLVMCodeGenTypes.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoDWARF.a  /opt/llvm/llvm-project/build/lib/libLLVMObject.a  /opt/llvm/llvm-project/build/lib/libLLVMIRReader.a  /opt/llvm/llvm-project/build/lib/libLLVMBitReader.a  /opt/llvm/llvm-project/build/lib/libLLVMAsmParser.a  /opt/llvm/llvm-project/build/lib/libLLVMCore.a  /opt/llvm/llvm-project/build/lib/libLLVMRemarks.a  /opt/llvm/llvm-project/build/lib/libLLVMBitstreamReader.a  /opt/llvm/llvm-project/build/lib/libLLVMMCParser.a  /opt/llvm/llvm-project/build/lib/libLLVMTextAPI.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoCodeView.a  /opt/llvm/llvm-project/build/lib/libLLVMNVPTXDesc.a  /opt/llvm/llvm-project/build/lib/libLLVMNVPTXInfo.a  /opt/llvm/llvm-project/build/lib/libLLVMMC.a  /opt/llvm/llvm-project/build/lib/libLLVMDebugInfoDWARFLowLevel.a  /opt/llvm/llvm-project/build/lib/libLLVMBinaryFormat.a  /opt/llvm/llvm-project/build/lib/libLLVMTargetParser.a  /opt/llvm/llvm-project/build/lib/libLLVMSupport.a  /opt/llvm/llvm-project/build/lib/libLLVMDemangle.a  -lrt  -ldl  -lm  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libzstd.so  /opt/llvm/llvm-project/build/lib/clang/23/lib/x86_64-unknown-linux-gnu/libflang_rt.runtime.a  /usr/lib/x86_64-linux-gnu/libffi.a && :
/opt/llvm/llvm-project/build/bin/clang-linker-wrapper: error: unable to find library -lomp
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
FAILED: runtimes/runtimes-stamps/runtimes-build /opt/llvm/llvm-project/build/runtimes/runtimes-stamps/runtimes-build 
cd /opt/llvm/llvm-project/build/runtimes/runtimes-bins && /opt/llvm/cmake-3.31.10-linux-x86_64/bin/cmake --build .
ninja: build stopped: subcommand failed.

The build failure seems to go away if we remove the arguments -DFLANG_RT_. However, dropping these gives me a compiler that gives the following error when enabling OpenMP:

/opt/llvm/llvm-project/install-step2/bin/clang-nvlink-wrapper: error: unable to find library -lompdevice
clang: error: nvlink command failed with exit code 1 (use -v to see invocation)
/opt/llvm/llvm-project/install-step2/bin/clang-linker-wrapper: error: 'clang' failed
flang-23: error: linker command failed with exit code 1 (use -v to see invocation)

So the -DFLANG_RT_ seems needed to produce the ompdevice library needed to actually do offloading.

Then we tried to manually do a 2-stage build, first building a host-compiler, with everything enabled except offloading, then using this compiler building an offloading-capable flang, but this got us stuch in the same -lomp error as before.

Does anyone have a “recipe” for building an offloading capable compiler, or any idea on what we do wrong?

2

@dreachem Do you have a working config you could share?

3

Right now there’s two ways to compile this, mine which builds off of the LLVM libc for GPUs and NVIDIA’s which builds off of CUDA support. For the former, you should be able to just use this cache file as a base llvm-project/offload/cmake/caches/FlangOffload.cmake at main · llvm/llvm-project · GitHub.

cmake ../llvm -G Ninja \
  -C ../offload/cmake/caches/FlangOffload.cmake \
  -DCMAKE_BUILD_TYPE=RELEASE \
  -DCMAKE_INSTALL_PREFIX=path/to/your/compiler

Your initial invocation seems to go for the CUDA support, but to actually use OpenMP on the GPU you need to build the OpenMP runtime targeting those architectures. The cache file should document how this is done. There’s some missing pieces here, mostly per-target support for module files. Also I think there’s some error that can show up with NVPTX erroneously thinking it has backtrace support, I’ll look into that later. Let me know if this works for you.

4

Thanks, that seems to have done the trick on several systems! Elegant and great!

I did however have a build failure on a single system:

[2678/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/stop.cpp.o
FAILED: flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/stop.cpp.o 
/opt/llvm/llvm-project/build-nvidia/bin/clang++ --target=nvptx64-nvidia-cuda -DLIBCXX_BUILDING_LIBCXXABI -DLIBC_NAMESPACE=__llvm_libc_common_utils -DLIBC_THREAD_MODE=LIBC_THREAD_MODE_SINGLE -D_GLIBCXX_NO_ASSERTIONS -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/opt/llvm/llvm-project/flang-rt/include -I/opt/llvm/llvm-project/flang-rt/../flang/include -I/opt/llvm/llvm-project/build-nvidia/runtimes/runtimes-nvptx64-nvidia-cuda-bins/flang-rt -I/opt/llvm/llvm-project/build-nvidia/include/nvptx64-nvidia-cuda/c++/v1 -I/opt/llvm/llvm-project/build-nvidia/include/c++/v1 -I/opt/llvm/llvm-project/libcxxabi/include -I/opt/llvm/llvm-project/cmake/Modules/../../libc -isystem /opt/llvm/llvm-project/build-nvidia/include/nvptx64-nvidia-cuda -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wno-comment -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-color -ffunction-sections -fdata-sections -Xclang -fno-pch-timestamp -O3 -DNDEBUG -std=gnu++17 -fno-exceptions -fno-rtti -funwind-tables -fno-asynchronous-unwind-tables "-D_GLIBCXX_THROW_OR_ABORT(_EXC)=(__builtin_abort())" -nogpulib -flto -fvisibility=hidden -Wno-unknown-cuda-version --cuda-feature=+ptx63 -Wp,-U_GLIBCXX_ASSERTIONS -Wp,-U_LIBCPP_ENABLE_ASSERTIONS -fdebug-prefix-map=/opt/llvm/llvm-project/build-nvidia/include/c++/v1=/opt/llvm/llvm-project/libcxx/include -nostdlibinc -nostdlib -nostdinc++ -MD -MT flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/stop.cpp.o -MF flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/stop.cpp.o.d -o flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/stop.cpp.o -c /opt/llvm/llvm-project/flang-rt/lib/runtime/stop.cpp
/opt/llvm/llvm-project/flang-rt/lib/runtime/stop.cpp:21:10: fatal error: 'execinfo.h' file not found
   21 | #include BACKTRACE_HEADER
      |          ^~~~~~~~~~~~~~~~
/opt/llvm/llvm-project/build-nvidia/runtimes/runtimes-nvptx64-nvidia-cuda-bins/flang-rt/config.h:22:26: note: expanded from macro 'BACKTRACE_HEADER'
   22 | #define BACKTRACE_HEADER <execinfo.h>
      |                          ^~~~~~~~~~~~
<scratch space>:28:1: note: expanded from here
   28 | <execinfo.h>
      | ^~~~~~~~~~~~
1 error generated.
[2679/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/type-info.cpp.o
[2680/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/assign.cpp.o
[2681/2707] Building CXX object libcxx/src/CMakeFiles/cxx_static.dir/algorithm.cpp.o
[2682/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/work-queue.cpp.o
[2683/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/type-code.cpp.o
[2684/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/complex-powi.cpp.o
[2685/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/temporary-stack.cpp.o
[2686/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/transformational.cpp.o
[2687/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/io-api-gpu.cpp.o
[2688/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/opt/llvm/llvm-project/flang/lib/Decimal/binary-to-decimal.cpp.o
[2689/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/product.cpp.o
[2690/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/format.cpp.o
[2691/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/sum.cpp.o
[2692/2707] Building CXX object libcxxabi/src/CMakeFiles/cxxabi_static_objects.dir/cxa_demangle.cpp.o
[2693/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/character.cpp.o
[2694/2707] Building CXX object libcxx/src/CMakeFiles/cxx_static.dir/locale.cpp.o
[2695/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/edit-output.cpp.o
[2696/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/reduction.cpp.o
[2697/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/reduce.cpp.o
[2698/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/tools.cpp.o
[2699/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/dot-product.cpp.o
[2700/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/matmul-transpose.cpp.o
[2701/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/matmul.cpp.o
[2702/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/extrema.cpp.o
[2703/2707] Building CXX object flang-rt/lib/runtime/CMakeFiles/flang_rt.runtime.static.dir/findloc.cpp.o
ninja: build stopped: subcommand failed.
[9657/9665] No patch step for 'runtimes'
FAILED: runtimes/runtimes-nvptx64-nvidia-cuda-stamps/runtimes-nvptx64-nvidia-cuda-build /opt/llvm/llvm-project/build-nvidia/runtimes/runtimes-nvptx64-nvidia-cuda-stamps/runtimes-nvptx64-nvidia-cuda-build 
cd /opt/llvm/llvm-project/build-nvidia/runtimes/runtimes-nvptx64-nvidia-cuda-bins && /opt/llvm/cmake-3.31.10-linux-x86_64/bin/cmake --build .
ninja: build stopped: subcommand failed.

This is git commit 6fae863 checked out this morning, so it is already including this recent PR that touch upon the same topic. I do not see exactly why it happens, but it was easy to work around, I just commented out completely the find_package(Backtrace) in flang-rt/CMakeLists.txt and now it builds fine. On another system, I did not see this failure… Strange.

5

I think it might depend on whether or not that machine has the relevant files for backtrace. The if-condition there is a little loose, since there’s technically several ways that you could get the NVPTX triple in, but if you’re using the cache file I’m a little surprised it apparently doesn’t work. I wonder if this should also be made to respect LLVM_ENABLE_BACKTRACE.

Does everything work with that aside?

6

As far as I see it, everything I expect to have working works on the systems I tried… Still some loose ends though, so that can change.

On the system where I have the backtrace-related failure, I do see /usr/include/execinfo.h - so the header file is there in a system-path but apparently not picked up correctly…

7

[flang-rt] Fix incorrect condition for removing backtrace (#184610) · llvm/llvm-project@0cbba3e · GitHub I think this was just an incredibly dumb logic error and I forgot to invert the condition.