Down with LD_LIBRARY_PATH, up with D101960

Today, to run an openmp application, one compiles it:

clang -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa hello.c -o a.out
execute,
./a.out
error while loading shared libraries: libomp.so: cannot open shared object file: No such file or directory

D101960, and this email, starts a discussion to remove that property of our toolchain. Today, the next steps go something like:

find $HOME -type f -iname libomp.so
and gets, on the machine I’m currently using,
$HOME/llvm-install/lib/libomp.so
$HOME/llvm-install-dbg/lib/libomp.so
$HOME/llvm-build-dbg/llvm/runtimes/runtimes-bins/openmp/runtime/src/libomp.so
$HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/runtime/src/libomp.so
$HOME/.emacs.d/lib/libomp.so
$HOME/.emacs.d/llvm/openmp/runtime/exports/lin_32e.ompt.optional/lib/libomp.so
$HOME/aomp/build/openmp/runtime/src/libomp.so
$HOME/aomp/build/openmp_debug/runtime/src/libomp.so
$HOME/rocm/aomp_13.0-3/lib/libomp.so
$HOME/rocm/aomp_13.0-3/lib-debug/libomp.so
$HOME/ROCm-llvm-build/llvm/lib/libomp.so

Brief confusion while I convince myself that I wanted the first one in that list,

export LD_LIBRARY_PATH=$HOME/llvm-install/lib/
./a.out
success

Similarly, if one takes a failing openmp test, it has a RUN line that can be copied into a shell, e.g.

$HOME/llvm-build/llvm/./bin/clang++ -fopenmp -fno-experimental-isel -I $HOME/llvm-project/openmp/libomptarget/test -I $HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/runtime/src -L $HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/libomptarget -L $HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/runtime/src -fopenmp-targets=amdgcn-amd-amdhsa $HOME/llvm-project/openmp/libomptarget/test/offloading/parallel_offloading_map.cpp -o $HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/libomptarget/test/amdgcn-amd-amdhsa/offloading/Output/parallel_offloading_map.cpp.tmp && $HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/libomptarget/test/amdgcn-amd-amdhsa/offloading/Output/parallel_offloading_map.cpp.tmp | $HOME/llvm-build/llvm/./bin/FileCheck $HOME/llvm-project/openmp/libomptarget/test/offloading/parallel_offloading_map.cpp

That will also fail, but for different reasons. Lit sets up LD_LIBRARY_PATH to find libomp + libomptarget, and LIBRARY_PATH to find the devicertl. The build tree puts the first two in different places, so the invocation to successfully hit the same failure from the test suite is something like

export LD_LIBRARY_PATH=“$HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/libomptarget:$HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/runtime/src”
export LIBRARY_PATH=“$HOME/llvm-build/llvm/runtimes/runtimes-bins/openmp/libomptarget/”
then the above.

Instead of this, let’s do something roughly equivalent to https://reviews.llvm.org/D101960.

That does a bunch of small, semi-orthogonal things with the end result that clang -fopenmp makes an executable that runs, without an environment variable, and the failures from the test suite can be copied into a new shell for debugging.

Thanks all,

Jon

I think your concern is part of a more profound problem.

At compile,

-fopenmp needs headers and -fopemmp-targets needs deviceRTL.
I think those under clang installation are picked by default.
I had not encountered an issue with this.

At link,
-fopenmp links libomp and -fopenmp-target needs libomptarget.
I reported an incident https://lists.llvm.org/pipermail/openmp-dev/2019-December/002990.html

I don’t see a good reason to pick up the one from OS instead of the one from clang installation if found.

Joel E. Denny attempted to tackle the issue but it went nowhere https://reviews.llvm.org/D55725
I would like to by default pick up the one from clang installation before looking for OS locations.
On the other hand, we should allow clang to accept libomp libomptarget from an alternative location.
Maybe we can have -fopenmp=libomp=/path/to/installation_root if an alternative searching location is desired.
Or -fopenmp=libompapi to turn on only compilation and requires setting all the include and library paths manually.
I mean more of the desired behaviors and please feel free to propose sane solutions.

At run,
LD_LIBRARY_PATH currently is needed to point out where libomp and libomptarget are.

As an application user, I’m unhappy if my apps stop because I forgot to load the llvm compiler module on an HPC to set LD_LIBRARY_PATH or my OS libomp got picked up for whatever reason. So I do support adding rpath for OpenMP libraries.
LD_LIBRARY_PATH has higher priority than rpath, using it to pick up alternative libraries remains working I believe.

With -fopenmp=libomp=/path/to/installation_root, we should be able to select alternative libraries at compiling and linking and with LD_LIBRARY_PATHw we can select dynamic libraries at running.

Best,

Ye