Using OpenMP target offloading in llvm-8.0.0

Runtimes OpenMP
1

Hi,

When trying to use openmp target offloading with llvm I get the following error

$ cat offload.cpp
#include <omp.h>
int main() {
#pragma omp target teams distribute parallel for
for(int i=0; i<100; i++);
return 0;
}
$ clang++ -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda --cuda-path=$CUDA_TOOLKIT_ROOT_DIR offload.cpp -o offload
$ ./offload
Libomptarget fatal error 1: default offloading policy must switched to mandatory or disabled
$

I have llvm-8.0.0 installed in my system at LLVM_PATH.
I downloaded openmp-8.0.0 source from the llvm download page. To build openmp I used the following command:

$ mkdir build && cd build
$ cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_INSTALL_PREFIX=$LLVM_PATH -DCMAKE_C_COMPILER=$LLVM_PATH/bin/clang -DCMAKE_CXX_COMPILER= $LLVM_PATH/bin/clang++ -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=35,60,70 …
$ make && make install
$

OpenMP got built without giving any error. But when trying to use target offloading with OpenMP I get the above mentioned fatal error. I tried this on 3 different machines, with same result.

2

Hi Alok,

What type of machine are you running on?

Can you try to see if any of the following env variables help in any way?

export OMP_DEFAULT_DEVICE=1
export CUDA_VISIBLE_DEVICES=0,1,2,3

Which OpenMP runtime are you using and are you picking up the correct one?

To resolve that use -I -L in the compile line:

Use -I to point to the dir containing omp.h header and -L to point to the OpenMP runtime library you’re aiming to use. Make sure that you don’t pick them from different installations.

Make sure that your LD_LIBRARY_PATH also contains the path to the OpenMP runtime library directory.

Thanks,

–Doru

3

Hi Doru,

I am running it on a linux cluster which has two NVIDIA P100-PCIE-16GB GPUs on its node. I also tried on an AWS machine with NVIDIA Tesla V100-SXM2-16GB GPU.

I tried setting OMP_DEFAULT_DEVICE=1 and CUDA_VISIBLE_DEVICES=0,1,2,3, but I’m getting the same error.

I’m using the same OpenMP runtime which I built.

My LD_LIBRARY_PATH is set to $LLVM_PATH/lib and $LLVM_PATH/lib64 (where both llvm-8 and openmp-8 are installed)
My C_INCLUDE_PATH and CXX_INCLUDE_PATH are set to $LLVM_PATH/include

4

HI Alok,

The two env vars I sent you before are just examples you should adapt them to whatever you have locally i.e. 2 GPUs or 1 GPU and set the OMP_DEFAULT_DEVICE accordingly 0,1, …

I see you built your OpenMP for multiple compute capabilities. Which one is the default? (there’s a cmake flag for that).

To make sure you always compile for the one you have on your system please use: -Xopenmp-target -march=sm_XX where XX can be 35, 60, 70 depending on the GPU you’re compiling for.

Thanks,

–Doru

5

Hi Doru,

Thanks a lot for the help. I was stuck on this issue for a couple of days. Setting -Xopenmp-target -march=sm_60 worked. :slight_smile: Now I am able to offload to GPU.

6

Excellent happy to hear! :slight_smile:

–Doru