clang-7: warning: No library 'libomptarget-nvptx-sm_35.bc'

Siegmar_Gross · July 2, 2018, 1:26pm

Hi,

today I've built the latest version of llvm-trunk using Cmake on my "SUSE
Linux Enterprise Server 12.3 (x86_64)". I get the following warnings, if I
compile a small program.

loki introduction 162 \clang --cuda-gpu-arch=sm_50 -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda dot_prod_accelerator_OpenMP.c -lomptarget
clang-7: warning: No library 'libomptarget-nvptx-sm_35.bc' found in the default clang lib directory or in LIBRARY_PATH. Expect degraded performance due to no inlining of runtime functions on target devices. [-Wopenmp-target]
clang-7: warning: argument unused during compilation: '--cuda-gpu-arch=sm_50' [-Wunused-command-line-argument]
loki introduction 163

Why does the compiler ignore "--cuda-gpu-arch=sm_50". How can I create
"libomptarget-nvptx-sm_35.bc" and probably "libomptarget-nvptx-sm_50.bc"?

Why do I get different numbers of devices, if I compile the program with
different options. I use a machine with two "Intel Xeon E5-2620 v3"
processors (6 cores and 12 hwthreads each) and one Nvidia Quadro K2200
device.

loki introduction 183 clang -fopenmp -fopenmp-targets=x86_64-pc-linux-gnu \
? dot_prod_accelerator_OpenMP.c -lomptarget
loki introduction 184 a.out
Number of processors: 24
Number of devices: 4
Default device: 0
sum = 6.000000e+08

loki introduction 185 clang -fopenmp -fopenmp-targets=nvptx64-nvidia-cuda \
? dot_prod_accelerator_OpenMP.c -lomptarget
clang-7: warning: No library 'libomptarget-nvptx-sm_35.bc' found in the default clang lib directory or in LIBRARY_PATH. Expect degraded performance due to no inlining of runtime functions on target devices. [-Wopenmp-target]
loki introduction 186 a.out
Number of processors: 24
Number of devices: 1
Default device: 0
sum = 6.000000e+08
loki introduction 187

Thank you very much for any answers in advance.

Kind regards

Siegmar

Gheorghe-Teod_Bercea · July 2, 2018, 3:10pm

Hi Siegmar,

It all depends on how you build the compiler.

If you want to build the .bc library you will need to have the following two cmake flags:

-DOPENMP_ENABLE_LIBOMPTARGET=ON
-DLIBOMPTARGET_NVPTX_ENABLE_BCLIB=true

This will enable the building of libomptarget library and the …sm_35.bc library. sm_35 is the default compute capability used by the compiler if you don’t specify anything else.

You can control this default value when you build your compiler by passing the following flag to your cmake command:

-DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_50

You can also build the .bc library for more than one compute capability using the following cmake flag:

-DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=50,60,70

Make sure to also include the default in the list (in this case 50).

When you compile your application/program you can control which compute capability the OpenMP device offloading toolchain uses by passing the following flag to clang/clang++:

-Xopenmp-target -march=sm_70

This compile line flag is only required if you want a compute capability other than the default (in this case sm_50) i.e. the one specified using the -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH flag.

I hope this helps,

–Doru

hahnjo · July 2, 2018, 4:29pm

Hi Siegmar,

please note that building the bclib requires Clang trunk itself, so you will have to rebuild the openmp repository with the "new" compiler. It doesn't work at all if you use GCC to compile the runtime libraries (which makes sense because GCC cannot generate LLVM Bitcode).

Why do I get different numbers of devices, if I compile the program with different options.

I think I already answered this in the past: libomptarget will show you the devices you compiled for. It creates 4 "virtual" devices for x86_64 and you probably have 1 GPU in your system.

Jonas

Siegmar_Gross · July 3, 2018, 8:30am

Hi Jonas, Gheorghe-Teod, Alexey,

thank you very much for your help. I tried to rebuild the current
version of llvm-trunk with clang-trunk from yesterday. Unfortunately,
I get an error.

loki build 128 clang -v
clang version 7.0.0 (trunk 336111)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/llvm-trunk/bin
Found candidate GCC installation: /usr/lib64/gcc/x86_64-suse-linux/4.8
Selected GCC installation: /usr/lib64/gcc/x86_64-suse-linux/4.8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda, version 9.0
loki build 129

loki build 129 cd ../llvm/
loki llvm 130 svn info --revision HEAD
Path: trunk
URL: http://llvm.org/svn/llvm-project/llvm/trunk
Relative URL: ^/llvm/trunk
Repository Root: http://llvm.org/svn/llvm-project
Repository UUID: 91177308-0d34-0410-b5e6-96231b3b80d8
Revision: 336173
Node Kind: directory
Last Changed Author: mkazantsev
Last Changed Rev: 336172
Last Changed Date: 2018-07-03 08:23:57 +0200 (Tue, 03 Jul 2018)

I use the following configure command. Hopefully, I don't have an error
in my command.

set LLVM_VERSION=llvm-trunk-new
set LLVM_COMPILER=llvm-trunk
rm -r build
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX:PATH=/usr/local/${LLVM_VERSION} \
   -DCMAKE_BUILD_TYPE:STRING="Release" \
   -DCMAKE_C_COMPILER:STRING="${DIRPREFIX_PROG}/${LLVM_COMPILER}/bin/clang" \
   -DCMAKE_C_FLAGS:STRING="-m64 -I/usr/local/valgrind/include -I/usr/include/ncurses" \
   -DCMAKE_C_STANDARD_LIBRARIES="-lpthread" \
   -DCMAKE_CXX_COMPILER:STRING="${DIRPREFIX_PROG}/${LLVM_COMPILER}/bin/clang++" \
   -DCMAKE_CXX_FLAGS:STRING="-m64 -I/usr/local/valgrind/include -I/usr/include/ncurses" \
   -DCMAKE_CXX_STANDARD_LIBRARIES="-lpthread" \
   -DCMAKE_EXE_LINKER_FLAGS:STRING="-m64" \
   -DCUDA_INCLUDE_DIRS:STRING="/usr/local/cuda/include" \
   -DCUDA_LIBRARIES:STRING="/usr/local/cuda/lib64/libcudart.so" \
   -DLLVM_PARALLEL_COMPILE_JOBS:STRING="8" \
   -DLLVM_PARALLEL_LINK_JOBS:STRING="8" \
   -DLLVM_LIBDIR_SUFFIX:STRING="64" \
   -DBUILD_SHARED_LIBS:BOOL=ON \
   -DLLVM_POLLY_LINK_INTO_TOOLS:BOOL=ON \
   -DLLVM_TARGETS_TO_BUILD:STRING="NVPTX;X86" \
   -DOPENMP_ENABLE_LIBOMPTARGET:BOOL=ON \
   -DLIBOMP_ENABLE_SHARED:BOOL=ON \
   -DLIBOMP_USE_HWLOC:BOOL=ON \
   -DLIBOMP_HWLOC_INSTALL_DIR:STRING="/usr/local/hwloc-2.0.1" \
   -DLIBOMPTARGET_DEP_LIBELF_INCLUDE_DIR:STRING="/usr/local/elfutils-0.169/include" \
   -DLIBOMPTARGET_DEP_LIBELF_LIBRARIES:STRING="/usr/local/elfutils-0.169/lib64/libelf.so" \
   -DLIBOMPTARGET_DEP_LIBFFI_INCLUDE_DIR:STRING="/usr/include" \
   -DLIBOMPTARGET_DEP_LIBFFI_LIBRARIES:STRING="/usr/lib64/libffi.so" \
   -DLIBOMPTARGET_NVPTX_ENABLE_BCLIB:BOOL=ON \
   -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES:STRING="50,60,70" \
   -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_50 \
   -DLIBOMPTARGET_DEBUG:BOOL=ON \
   -GNinja \
   ../llvm \
   >& tee log.cmake

"log.cmake" contains the following lines.

loki build 135 more log.cmake
-- The C compiler identification is Clang 7.0.0
-- The CXX compiler identification is Clang 7.0.0
-- The ASM compiler identification is Clang
-- Found assembler: /usr/local/llvm-trunk/bin/clang
-- Check for working C compiler using: Ninja
-- Check for working C compiler using: Ninja -- works
...
-- Could NOT find Python module pygments
-- Could NOT find Python module pygments.lexers.c_cpp
-- Could NOT find Python module yaml
...
-- Found PythonInterp: /usr/bin/python2.7 (found version "2.7.13")
...
-- Compiler-RT supported architectures: x86_64
-- Looking for rpc/xdr.h
-- Looking for rpc/xdr.h - not found
-- Looking for tirpc/rpc/xdr.h
-- Looking for tirpc/rpc/xdr.h - not found
...
-- Found LIBOMPTARGET_DEP_LIBELF: /usr/local/elfutils-0.169/lib64/libelf.so
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.28")
-- Found LIBOMPTARGET_DEP_LIBFFI: /usr/lib64/libffi.so
-- Found LIBOMPTARGET_DEP_CUDA_DRIVER: /usr/lib64/libcuda.so
-- LIBOMPTARGET: Building offloading runtime library libomptarget.
-- LIBOMPTARGET: Not building aarch64 offloading plugin: machine not found in the system.
-- LIBOMPTARGET: Building CUDA offloading plugin.
-- LIBOMPTARGET: Not building PPC64 offloading plugin: machine not found in the system.
-- LIBOMPTARGET: Not building PPC64le offloading plugin: machine not found in the system.
-- LIBOMPTARGET: Building x86_64 offloading plugin.
-- LIBOMPTARGET: Building CUDA offloading device RTL.
-- Performing Test LIBOMPTARGET_NVPTX_CUDA_COMPILER_SUPPORTS_FLAGS_REQUIRED
-- Performing Test LIBOMPTARGET_NVPTX_CUDA_COMPILER_SUPPORTS_FLAGS_REQUIRED - Success
-- Performing Test LIBOMPTARGET_NVPTX_CUDA_COMPILER_SUPPORTS_EXTERN_SHARED
-- Performing Test LIBOMPTARGET_NVPTX_CUDA_COMPILER_SUPPORTS_EXTERN_SHARED - Failed
-- Performing Test LIBOMPTARGET_NVPTX_CUDA_COMPILER_SUPPORTS_FCUDA_RDC
-- Performing Test LIBOMPTARGET_NVPTX_CUDA_COMPILER_SUPPORTS_FCUDA_RDC - Success
-- LIBOMPTARGET: Building CUDA LLVM bitcode offloading device RTL.
-- ISL version: isl-0.19-185-g8e9f55ce
-- Performing Test HAS_ATTRIBUTE_WARN_UNUSED_RESULT
-- Performing Test HAS_ATTRIBUTE_WARN_UNUSED_RESULT - Success
...
-- Performing Test LLDB_USING_LIBSTDCXX_4_9
-- Performing Test LLDB_USING_LIBSTDCXX_4_9 - Failed
CMake Warning at tools/lldb/cmake/modules/LLDBConfig.cmake:402 (message):
   You appear to be linking to libstdc++ version lesser than 4.9 without
   exceptions enabled. These versions of the library have an issue, which
   causes occasional lldb crashes. See
   <https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59656> for details. Possible
   courses of action are:

- use libstdc++ version 4.9 or newer

- use libc++ (via LLVM_ENABLE_LIBCXX)

- enable exceptions (via LLVM_ENABLE_EH)

- ignore this warning and accept occasional instability
Call Stack (most recent call first):
tools/lldb/CMakeLists.txt:11 (include)
...
-- Found libedit: /usr/include (found version ".")
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.6")
-- Found SWIG: /usr/local/swig-3.0.12/bin/swig (found version "3.0.12")
-- Performing Test CXX_SUPPORTS_NO_MACRO_REDEFINED
-- Performing Test CXX_SUPPORTS_NO_MACRO_REDEFINED - Success
-- Symbols (liblldb): exporting all symbols from the lldb namespace
-- Configuring done
-- Generating done
-- Build files have been written to: /export2/src/llvm-trunk/build
loki build 136

Why don't you find the Python modules and xdr.h? I ask our admin to upgrade
libstdc++.

loki build 138 ls -ld /usr/lib64/python2.7/site-packages/yaml/
drwxr-xr-x 2 root root 4096 May 11 06:12 /usr/lib64/python2.7/site-packages/yaml/

loki build 139 ls -ld /usr/lib/python2.7/site-packages/pygments
drwxr-xr-x 6 root root 4096 Jul 31 2017 /usr/lib/python2.7/site-packages/pygments

loki build 140 ls -l /usr/include/rpc/xdr.h
-rw-r--r-- 1 root root 14577 May 28 20:07 /usr/include/rpc/xdr.h

Building the package breaks with the following error.

loki build 145 grep FAILED log.ninja-build
FAILED: include/llvm/IR/IntrinsicImpl.inc.tmp
FAILED: lib/ToolDrivers/llvm-lib/Options.inc.tmp
FAILED: tools/clang/include/clang/Driver/Options.inc.tmp
FAILED: tools/lld/lib/Driver/DarwinLdOptions.inc.tmp
FAILED: lib/Transforms/InstCombine/InstCombineTables.inc.tmp
FAILED: include/llvm/IR/IntrinsicEnums.inc.tmp
FAILED: include/llvm/IR/Attributes.inc.tmp
FAILED: tools/lld/ELF/Options.inc.tmp
FAILED: tools/lld/COFF/Options.inc.tmp
FAILED: lib/IR/AttributesCompatFunc.inc.tmp
FAILED: lib/ToolDrivers/llvm-dlltool/Options.inc.tmp
FAILED: tools/llvm-rc/Opts.inc.tmp
FAILED: tools/llvm-cvtres/Opts.inc.tmp
FAILED: tools/lld/wasm/Options.inc.tmp
FAILED: tools/lld/MinGW/Options.inc.tmp
FAILED: tools/llvm-objcopy/ObjcopyOpts.inc.tmp
FAILED: tools/llvm-mt/Opts.inc.tmp
loki build 146 grep error: log.ninja-build
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
/export2/src/llvm-trunk/build/bin/llvm-tblgen: symbol lookup error: /export2/src/llvm-trunk/build/bin/llvm-tblgen: undefined symbol: _ZTVN4llvm2cl3optISsLb0ENS0_6parserISsEEEE
loki build 147

Hopefully somebody can fix the problem. Thank you very much for any
help in advance.

Kind regards

Siegmar

hahnjo · July 3, 2018, 10:34am

These look like errors in LLVM, not the OpenMP runtime. As I wrote in my last message you *only* need to build the *openmp* repository with Clang trunk, no the entire LLVM system.

In general you shouldn't be too worried if the build system can't find optional dependencies: It's CMake's job to figure out what it can do. IIRC not finding some python modules will only disable some tests (btw I don't see why updating libstdc++ should help here?!?).
Regarding you CMake invocation: You generally don't need shared libraries of LLVM and Clang for release builds, static libraries are just fine without debug symbols.

Regards,
Jonas