OpenMP NVidia offload build problem 13.0.1?

I hope I’m in the right place. The documentation at openmp.llvm.org says to contact openmp-dev@lists.llvm.org but that email says I’m not allowed to post to the list and it looks like maybe the list is obsolete anyway?

I can only find documentation that seems to be for 15.0.0 at Support, Getting Involved, and FAQ — LLVM/OpenMP 15.0.0git documentation so this problem is based on me trying to use that even though I’m trying to build 13.0.1. If there’s 13.0.1 documentation please point me to it. (I also looked at Building LLVM/Clang with OpenMP Offloading to NVIDIA GPUs - HPC Wiki but that seems old and incomplete)

LLVM 13.0.1
CentOS 7.9
GCC 11.2.0
cmake 3.18.2
the build host doesn’t not have GPUs and/or any NVidia software installed on it.

The openmp.llvm.org documentation says…

Q: How to build an OpenMP GPU offload capable compiler?

To build an effective OpenMP offload capable compiler, only one extra CMake option, 
LLVM_ENABLE_RUNTIMES=”openmp”, is needed when building LLVM (Generic information 
about building LLVM is available here.). Make sure all backends that are targeted by OpenMP 
to be enabled. By default, Clang will be built with all backends enabled. When building with 
LLVM_ENABLE_RUNTIMES=”openmp” OpenMP should not be enabled in 
LLVM_ENABLE_PROJECTS because it is enabled by default.

For NVidia offload, please see Q: How to build an OpenMP NVidia offload capable compiler?. 
For AMDGPU offload, please see Q: How to build an OpenMP AMDGPU offload capable compiler?.

…and…

Q: How to build an OpenMP NVidia offload capable compiler?

The Cuda SDK is required on the machine that will execute the openmp application.

If your build machine is not the target machine or automatic detection of the available
GPUs failed, you should also set:

    CLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_XX where XX is the architecture of 
    your GPU, e.g, 80.
    LIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=YY where YY is the numeric 
    compute capacity of your GPU, e.g., 75.

… but if I build LLVM without “openmp” in LLVM_ENABLE_PROJECTS, then cmake decides that openmp is “disabled”

cmake                                                                          \
-DCMAKE_C_COMPILER=${HOST_GCC}/bin/gcc                                         \
-DCMAKE_CXX_COMPILER=${HOST_GCC}/bin/g++                                       \
-DGCC_INSTALL_PREFIX=${HOST_GCC}                                               \
-DLLVM_ENABLE_PROJECTS="clang;compiler-rt;libclc;libcxx;libcxxabi;libunwind;lld;lldb;polly;clang-tools-extra;flang;parallel-libs;pstl" \
-DCMAKE_CXX_LINK_FLAGS="-L${HOST_GCC}/lib64 -Wl,-rpath,${HOST_GCC}/lib64"      \
-DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX}                                       \
-DLLVM_ENABLE_ASSERTIONS=ON                                                    \
-DCMAKE_BUILD_TYPE="Release"                                                   \
-DLLVM_TARGETS_TO_BUILD="X86"                                                  \
-DLLVM_ENABLE_RUNTIMES="openmp" -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_70 -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=70 \
../llvm

as in…

  .
  .
  .
-- lldb project is enabled
-- mlir project is enabled
-- openmp project is disabled
-- parallel-libs project is enabled
-- polly project is enabled
  .
  .
  .

…and there’s no indication that cmake pays any attention to the other openmp related flags to enable openmp in the build.

On the other hand, if I add ‘openmp’ to the LLVM_ENABLE_PROJECTS then things (at first) appear better but eventually there are CMake errors that the configuration is “incomplete”.

  .
  .
  .
-- lldb project is enabled
-- mlir project is enabled
-- openmp project is enabled
-- parallel-libs project is enabled
-- polly project is enabled
  .
  .
  .
-- sanitizer_common tests on "Linux" will run against "asan;lsan;msan;tsan;ubsan"
-- check-shadowcallstack does nothing.
-- Performing Test OPENMP_HAVE_WALL_FLAG
-- Performing Test OPENMP_HAVE_WALL_FLAG - Success
-- Performing Test OPENMP_HAVE_WERROR_FLAG
-- Performing Test OPENMP_HAVE_WERROR_FLAG - Success
-- Performing Test OPENMP_HAVE_WCAST_QUAL_FLAG
-- Performing Test OPENMP_HAVE_WCAST_QUAL_FLAG - Success
-- Performing Test OPENMP_HAVE_WFORMAT_PEDANTIC_FLAG
-- Performing Test OPENMP_HAVE_WFORMAT_PEDANTIC_FLAG - Failed
-- Performing Test OPENMP_HAVE_WIMPLICIT_FALLTHROUGH_FLAG
-- Performing Test OPENMP_HAVE_WIMPLICIT_FALLTHROUGH_FLAG - Success
-- Performing Test OPENMP_HAVE_WSIGN_COMPARE_FLAG
-- Performing Test OPENMP_HAVE_WSIGN_COMPARE_FLAG - Success
-- Performing Test OPENMP_HAVE_WNO_EXTRA_FLAG
-- Performing Test OPENMP_HAVE_WNO_EXTRA_FLAG - Success
-- Performing Test OPENMP_HAVE_WNO_PEDANTIC_FLAG
-- Performing Test OPENMP_HAVE_WNO_PEDANTIC_FLAG - Success
-- Performing Test OPENMP_HAVE_WNO_MAYBE_UNINITIALIZED_FLAG
-- Performing Test OPENMP_HAVE_WNO_MAYBE_UNINITIALIZED_FLAG - Success
-- Performing Test OPENMP_HAVE_STD_CPP14_FLAG
-- Performing Test OPENMP_HAVE_STD_CPP14_FLAG - Success
-- Performing Test LIBOMP_HAVE_FNO_EXCEPTIONS_FLAG
-- Performing Test LIBOMP_HAVE_FNO_EXCEPTIONS_FLAG - Success
-- Performing Test LIBOMP_HAVE_FNO_RTTI_FLAG
-- Performing Test LIBOMP_HAVE_FNO_RTTI_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_CLASS_MEMACCESS_FLAG
-- Performing Test LIBOMP_HAVE_WNO_CLASS_MEMACCESS_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_COVERED_SWITCH_DEFAULT_FLAG
-- Performing Test LIBOMP_HAVE_WNO_COVERED_SWITCH_DEFAULT_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_FRAME_ADDRESS_FLAG
-- Performing Test LIBOMP_HAVE_WNO_FRAME_ADDRESS_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_STRICT_ALIASING_FLAG
-- Performing Test LIBOMP_HAVE_WNO_STRICT_ALIASING_FLAG - Success
-- Performing Test LIBOMP_HAVE_WSTRINGOP_OVERFLOW_FLAG
-- Performing Test LIBOMP_HAVE_WSTRINGOP_OVERFLOW_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_STRINGOP_TRUNCATION_FLAG
-- Performing Test LIBOMP_HAVE_WNO_STRINGOP_TRUNCATION_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_SWITCH_FLAG
-- Performing Test LIBOMP_HAVE_WNO_SWITCH_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_UNINITIALIZED_FLAG
-- Performing Test LIBOMP_HAVE_WNO_UNINITIALIZED_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_UNUSED_BUT_SET_VARIABLE_FLAG
-- Performing Test LIBOMP_HAVE_WNO_UNUSED_BUT_SET_VARIABLE_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_RETURN_TYPE_C_LINKAGE_FLAG
-- Performing Test LIBOMP_HAVE_WNO_RETURN_TYPE_C_LINKAGE_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_CAST_QUAL_FLAG
-- Performing Test LIBOMP_HAVE_WNO_CAST_QUAL_FLAG - Success
-- Performing Test LIBOMP_HAVE_WNO_INT_TO_VOID_POINTER_CAST_FLAG
-- Performing Test LIBOMP_HAVE_WNO_INT_TO_VOID_POINTER_CAST_FLAG - Success
-- Performing Test LIBOMP_HAVE_MSSE2_FLAG
-- Performing Test LIBOMP_HAVE_MSSE2_FLAG - Success
-- Performing Test LIBOMP_HAVE_FTLS_MODEL_FLAG
-- Performing Test LIBOMP_HAVE_FTLS_MODEL_FLAG - Success
-- Performing Test LIBOMP_HAVE_MMIC_FLAG
-- Performing Test LIBOMP_HAVE_MMIC_FLAG - Failed
-- Performing Test LIBOMP_HAVE_M32_FLAG
-- Performing Test LIBOMP_HAVE_M32_FLAG - Failed
-- Looking for shm_open
-- Looking for shm_open - not found
-- Looking for shm_open
-- Looking for shm_open - found
-- Performing Test LIBOMP_HAVE_X_FLAG
-- Performing Test LIBOMP_HAVE_X_FLAG - Success
-- Performing Test LIBOMP_HAVE_WARN_SHARED_TEXTREL_FLAG
-- Performing Test LIBOMP_HAVE_WARN_SHARED_TEXTREL_FLAG - Success
-- Performing Test LIBOMP_HAVE_AS_NEEDED_FLAG
-- Performing Test LIBOMP_HAVE_AS_NEEDED_FLAG - Success
-- Performing Test LIBOMP_HAVE_VERSION_SCRIPT_FLAG
-- Performing Test LIBOMP_HAVE_VERSION_SCRIPT_FLAG - Success
-- Performing Test LIBOMP_HAVE_STATIC_LIBGCC_FLAG
-- Performing Test LIBOMP_HAVE_STATIC_LIBGCC_FLAG - Success
-- Performing Test LIBOMP_HAVE_Z_NOEXECSTACK_FLAG
-- Performing Test LIBOMP_HAVE_Z_NOEXECSTACK_FLAG - Success
-- Looking for immintrin.h
-- Looking for immintrin.h - found
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_RTM
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_RTM - Success
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_WAITPKG
-- Performing Test LIBOMP_HAVE_ATTRIBUTE_WAITPKG - Success
-- Performing Test LIBOMP_HAVE_WAITPKG_INTRINSICS
-- Performing Test LIBOMP_HAVE_WAITPKG_INTRINSICS - Success
-- Performing Test LIBOMP_HAVE_RTM_INTRINSICS
-- Performing Test LIBOMP_HAVE_RTM_INTRINSICS - Success
-- Found Perl: /usr/bin/perl (found version "5.16.3")
-- Performing Test LIBOMP_HAVE_VERSION_SYMBOLS
-- Performing Test LIBOMP_HAVE_VERSION_SYMBOLS - Success
-- Performing Test LIBOMP_HAVE___BUILTIN_FRAME_ADDRESS
-- Performing Test LIBOMP_HAVE___BUILTIN_FRAME_ADDRESS - Success
-- Performing Test LIBOMP_HAVE_WEAK_ATTRIBUTE
-- Performing Test LIBOMP_HAVE_WEAK_ATTRIBUTE - Success
-- Performing Test LIBOMP_HAVE_PSAPI
-- Performing Test LIBOMP_HAVE_PSAPI - Failed
-- Looking for sqrt in m
-- Looking for sqrt in m - found
-- Looking for __atomic_load_1
-- Looking for __atomic_load_1 - not found
-- Looking for __atomic_load_1 in atomic
-- Looking for __atomic_load_1 in atomic - found
-- Using LLVM include directories: /usr/src/llvm-project-13.0.1.src/llvm/include;/usr/src/llvm-project-13.0.1.src/build.ucs3/include
-- Found LIBOMPTARGET_DEP_LIBELF: /usr/lib64/libelf.so  
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.27.1")
-- Found LIBOMPTARGET_DEP_LIBFFI: /usr/lib64/libffi.so  
-- Could NOT find LIBOMPTARGET_DEP_CUDA_DRIVER (missing: LIBOMPTARGET_DEP_CUDA_DRIVER_LIBRARIES)
-- Could NOT find LIBOMPTARGET_DEP_VEO (missing: LIBOMPTARGET_DEP_VEO_LIBRARIES LIBOMPTARGET_DEP_VEOSINFO_LIBRARIES LIBOMPTARGET_DEP_VEO_INCLUDE_DIRS)
-- LIBOMPTARGET: Building offloading runtime library libomptarget.
-- LIBOMPTARGET: Not building aarch64 offloading plugin: machine not found in the system.
-- LIBOMPTARGET: Building AMDGPU plugin for dlopened libhsa
-- LIBOMPTARGET: Not generating amdgcn test targets as amdgpu-arch is not found
-- LIBOMPTARGET: Building CUDA offloading plugin.
-- LIBOMPTARGET: Building CUDA plugin for dlopened libcuda
-- LIBOMPTARGET: Disabling tests using CUDA plugin as cuda may not be available
-- LIBOMPTARGET: Not building PPC64 offloading plugin: machine not found in the system.
-- LIBOMPTARGET: Not building PPC64le offloading plugin: machine not found in the system.
-- LIBOMPTARGET: Not building nec-aurora plugin: libveo or libveosinfo not found.
-- LIBOMPTARGET: Building x86_64 offloading plugin.
-- LIBOMPTARGET: Building AMDGCN device RTL. Using clang from in-tree build
-- LIBOMPTARGET: Not building NVPTX deviceRTL by default on CUDA free system.
-- LIBOMPTARGET: Not building NVPTX deviceRTL by default on CUDA free system.
-- LIBOMPTARGET: Building the llvm-omp-device-info tool
-- Looking for __atomic_load_1
-- Looking for __atomic_load_1 - not found
-- Looking for __atomic_load_1 in atomic
-- Looking for __atomic_load_1 in atomic - found
-- Looking for sys/resource.h
-- Looking for sys/resource.h - found
-- Clang version: 13.0.1
  .
  .
  .
-- Performing Test HAVE_SYS_INTTYPES_H
-- Performing Test HAVE_SYS_INTTYPES_H - Failed
-- Registering Polly as a pass plugin (static build: ON)
CMake Error at cmake/modules/LLVMExternalProjectUtils.cmake:358 (add_custom_target):
  add_custom_target cannot create target "check-openmp" because another
  target with the same name already exists.  The existing target is a custom
  target created in source directory
  "/usr/src/llvm-project-13.0.1.src/openmp".  See documentation for
  policy CMP0002 for more details.
Call Stack (most recent call first):
  runtimes/CMakeLists.txt:226 (llvm_ExternalProject_Add)
  runtimes/CMakeLists.txt:363 (runtime_default_target)


-- Registering Bye as a pass plugin (static build: OFF)
-- LLVM FileCheck Found: /usr/bin/FileCheck
  .
  .
  .
-- Performing Test HAVE_SYS_INTTYPES_H
-- Performing Test HAVE_SYS_INTTYPES_H - Failed
-- Performing Test HAVE_STEADY_CLOCK
-- Performing Test HAVE_STEADY_CLOCK -- success
-- Configuring incomplete, errors occurred!
See also "/usr/src/llvm-project-13.0.1.src/build.ucs3/CMakeFiles/CMakeOutput.log".
See also "/usr/src/llvm-project-13.0.1.src/build.ucs3/CMakeFiles/CMakeError.log".

Something else that I find unclear from the documentation is that I’m trying to build a single install that supports both host OpenMP (i.e. the usual OpenMP stuff) and NVidia OpenMP support. (We NFS mount tools on different machines from a single server). So, do I need to somehow specify multiple things so the host and NVidia are both supported or can I only have one or the other and have to make separate installs for different machines.

Thanks.

LLVM’s OpenMP target offloading uses a bitcode runtime library for the device routines. This is generally only expected to be compiled with the same Clang that will be used to create the final application. When building with the LLVM_ENABLE_RUNTIMES=openmp method the OpenMP project will be build automatically with the compiler that was just created. For this to work, the LLVM_ENABLE_PROJECTS option should not be used otherwise the CMake targets will conflict. From looking at your error message it seems like you either have both enabled, or you did not clear your cache when trying the runtimes option. Also it shows that it is not building the CUDA support because CUDA was not found on the system, so you will need a valid CUDA installation and it should be picked up automatically. Try deleting your CMake cache and running your configuration again.