OpenMP nvptx64 offload fails silently with clang 14.0.3 but works with trunk (17.0.0)

Runtimes OpenMP
1

Hello

Trying to build OpenMP nvptx64 offloading with a clang 14.0.3 dependency. I’m building and bootstrapping clang, then use it to compile a test program which includes a printf to report whether or not it runs on the GPU. Compilation succeeds without errors or warnings, yet it will not execute on the GPU. I have tried the same against llvm trunk, it works fine and I can see my test program being executed on the GPU. Is there a way to tell why it does not attempt to execute on the GPU, or how can I get it to do that?

To reproduce:

Ubuntu 22.04.1 LTS
clang 17.0.0 (hash 7a49d50f22ad577d91cda7904c8a162c2cecd4a8)
clang 14.0.3 (hash 1f9140064dfbfb0bbda8e51306ea51080b2f7aac)

nproc=20

# first build
mkdir build
cd build
cmake ../llvm/ -DCMAKE_BUILD_TYPE=Release -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_86 -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=86 -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DLLVM_ENABLE_PROJECTS="clang;openmp"
make -j$(nproc)

# bootstrap build
cd ..
mkdir build2
cd build2
CC=../build/bin/clang CXX=../build/bin/clang++ cmake ../llvm/ -DCMAKE_BUILD_TYPE=Release -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_86 -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=86 -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DLLVM_ENABLE_PROJECTS="clang;openmp" -DCMAKE_INSTALL_PREFIX=$HOME/.local
make -j$(nproc)
make install

mkdir clangtest
cd clangtest
cp ..../main.cpp ./
PATH=$HOME/.local/bin:$PATH LD_LIBRARY_PATH=$HOME/.local/lib $HOME/.local/bin/clang++ -fopenmp -fopenmp-targets=nvptx64 -O3 main.cpp
PATH=$HOME/.local/bin:$PATH LD_LIBRARY_PATH=$HOME/.local/lib ./a.out
2

I’m assuming using trunk Clang isn’t a solution here? I’m not sure what could be going wrong with your 14.0.3 build. Could you build the 14.0.3 compiler with -DLIBOMPTARGET_ENABLE_DEBUG=ON and provide the output when you run your application with env LIBOMPTARGET_DEBUG=1 ./your-app. Also, printf support can be a little weird in certain cases. The following program should be sufficient to check for offloading with OpenMP.

#include <omp.h>

int main() {
  int isDevice = 0;
#pragma omp target map(from : isDevice)
  { isDevice = omp_is_initial_device(); }
  return isDevice;
}

$ clang test.c -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda -Xopenmp-target=nvptx64-nvidia-cuda -march=sm_86
$ ./a.out && echo "Offloaded!"

Also, -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_86 was removed in 16.0.

3

Thanks for the reply.
When I try that, unfortunately “Offloaded” is not printed. Here’s the output I get:

$ LIBOMPTARGET_DEBUG=1 ./a.out && echo "Offloaded!"
Libomptarget --> Init target library!
Libomptarget --> Loading RTLs...
Libomptarget --> Loading library 'libomptarget.rtl.ppc64.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.ppc64.so': libomptarget.rtl.ppc64.so: cannot open shared object file: No such file or directory!
Libomptarget --> Loading library 'libomptarget.rtl.x86_64.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.x86_64.so': libomptarget.rtl.x86_64.so: cannot open shared object file: No such file or directory!
Libomptarget --> Loading library 'libomptarget.rtl.cuda.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.cuda.so': libomptarget.rtl.cuda.so: cannot open shared object file: No such file or directory!
Libomptarget --> Loading library 'libomptarget.rtl.aarch64.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.aarch64.so': libomptarget.rtl.aarch64.so: cannot open shared object file: No such file or directory!
Libomptarget --> Loading library 'libomptarget.rtl.ve.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.ve.so': libomptarget.rtl.ve.so: cannot open shared object file: No such file or directory!
Libomptarget --> Loading library 'libomptarget.rtl.amdgpu.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.amdgpu.so': libomptarget.rtl.amdgpu.so: cannot open shared object file: No such file or directory!
Libomptarget --> Loading library 'libomptarget.rtl.rpc.so'...
Libomptarget --> Unable to load library 'libomptarget.rtl.rpc.so': libomptarget.rtl.rpc.so: cannot open shared object file: No such file or directory!
Libomptarget --> RTLs loaded!
Libomptarget --> No RTL found for image 0x0000000000402080!
Libomptarget --> Done registering entries!
Libomptarget --> Entering target region with entry point 0x0000000000402008 and device Id -1
Libomptarget --> Call to omp_get_num_devices returning 0
Libomptarget --> Default TARGET OFFLOAD policy is now disabled (no devices were found)
Libomptarget --> Offload is disabled
Libomptarget --> Not offloading to device -1
Libomptarget --> Unloading target library!
Libomptarget --> No RTLs in use support the image 0x0000000000402080!
Libomptarget --> Done unregistering images!
Libomptarget --> Translation table for descriptor 0x0000000000422340 cannot be found, probably it has been already removed.
Libomptarget --> Done unregistering library!
Libomptarget --> Deinit target library!
$
4

It’s not finding the CUDA RTL. Make sure it was built by checking for libomptarget.rtl.cuda.so and make sure it’s in your library path. Something like
export LD_LIBRARY_PATH=${PATH_TO_LLVM_INSTALL}/lib:${LD_LIBRARY_PATH}

1 Like
5

I can see libomptarget.rtl.cuda.so being built using 17.0.0 but not using 14.0.3. Specifically, both versions build

  • libomp.so
  • libompd.so
  • libomptarget.so
  • libomptarget(-new)?-nvptx-sm_86.bc

However, only clang 17.0.0 will build the libomptarget.rtl.(cuda | x86_64 | amdgpu).so libs.

Does 14.0.3 need any additional config or flags to get it to build the rtl library? I’ve tried changing the flags of the second build from LLVM_ENABLE_PROJECTS="clang;openmp" (as stated above) to LLVM_ENABLE_PROJECTS="clang" LLVM_ENABLE_RUNTIMES="openmp". However, this did not trigger a build of the rtl libs.

6

-- LIBOMPTARGET: Not building CUDA offloading plugin: libelf dependency not found.

Apparently this is needed as an external dependency by clang 14.0.3, quick install with apt and offloading works as expected. Thanks for the help!

7

We replaced that dependency with LLVM’s ELF library in 15.0. Glad you got it figured out.

1 Like