AMDGPU and clang-tools-extra

Date: Wed, 19 May 2021 15:09:18 +0900
From: Itaru Kitayama via Openmp-dev <openmp-dev@lists.llvm.org>
To: openmp-dev <openmp-dev@lists.llvm.org>
Subject: [Openmp-dev] AMDGPU and clang-tools-extra
Message-ID:
<CANW9uysd4uCQ7j3Jr1qmn2=RPo5a5OGU9e8-vPuuGv=P0nAXDQ@mail.gmail.com>
Content-Type: text/plain; charset=“UTF-8”

I am trying to build llvm on main branch with Spack with help from Tom, however
the AMDGPU code does not seem to work well with clang-tools-extra
project, which
is added together with the clang project when a Spack build initiated.
Has anyone
observed an issue with AMDGPU code with tools in upstream?

None reported. I don’t build clang-tools-extra, seems totally independent.

Are you building openmp with ENABLE_PROJECTS and a clang that can’t build amdgpu code, instead of ENABLE_RUNTIMES? If so, use ENABLE_RUNTIMES, a working clang, or disable the amdgpu library.

Otherwise, please expand on ‘does not seem to work well’.

Jon

spack install llvm@main does work on my workstation with ubuntu 20.04. rocm is installed and amd pieces for OpenMP offload are rebuilt properly. I have limited ideas about the experience on other OSes.

yeluo@epyc-server:~/opt/llvm-clang/llvm-project$ ls /home/packages/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-main-6n25gihrin6rc727fbothf44yq6zbp5a/lib/libomp
libomp.so libomptarget-amdgcn-gfx803.bc libomptarget.rtl.cuda.so
libomptarget-amdgcn-gfx700.bc libomptarget-amdgcn-gfx900.bc libomptarget.rtl.x86_64.so
libomptarget-amdgcn-gfx701.bc libomptarget-amdgcn-gfx906.bc libomptarget.so
libomptarget-amdgcn-gfx801.bc libomptarget.rtl.amdgpu.so

yeluo@epyc-server:~/opt/llvm-clang/llvm-project$ which clang-tidy
/home/packages/spack/opt/spack/linux-ubuntu20.04-zen2/gcc-10.2.0/llvm-main-6n25gihrin6rc727fbothf44yq6zbp5a/bin/clang-tidy

spack uses ENABLE_PROJECTS and doesn’t use ENABLE_RUNTIMES right now.
So the output from building clang-tools-extra and openmp can be blended.

Best,

Ye

Jon, Ye,
Is this

-DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="openmp"

correct? I used, and Spack as well, combined them (clang and openmp
projects) together indeed in ENABLE_PROJECTS.

Via enable projects will build openmp using the compiler you use for llvm. Via enable runtimes will build clang, and then use that clang to build openmp. You might want either.

Shilei was/is looking at making the compiler checking more precise than ‘is it called clang’ which will convert “doesn’t build” problems into “looked like it worked but there is no openmp”.

Thanks!

-DLLVM_ENABLE_PROJECTS=“clang” -DLLVM_ENABLE_RUNTIMES=“openmp”
I think this is what Jon refers to.

If you have issues building llvm@main in spack. Please provide your spack command and attach the following files

spack-build-01-cmake-out.txt
spack-build-02-build-out.txt
from your spack-stage llvm build folder.
Best,

Ye

Hi all,
As the files are too large for the list; I've uploaded them to:

https://github.com/spack/spack/issues/23654

Based on the CMake output, LLVM was found on the machine so LLVM_DIR is set.
However, clang was not found so CLANG_TOOL is NOT-FOUND

if (LLVM_DIR)

Builds that use pre-installed LLVM have LLVM_DIR set.

find_program(CLANG_TOOL clang PATHS ${LLVM_TOOLS_BINARY_DIR} NO_DEFAULT_PATH)
find_program(LINK_TOOL llvm-link PATHS ${LLVM_TOOLS_BINARY_DIR}
NO_DEFAULT_PATH)
find_program(OPT_TOOL opt PATHS ${LLVM_TOOLS_BINARY_DIR} NO_DEFAULT_PATH)

libomptarget_say(“Building AMDGCN device RTL. Using clang: ${CLANG_TOOL}”)

Use ENABLE_RUNTIMES does workaround this issue as this part is bypassed.

@Jon please take care of the case when any of the find_program fails.

Second issue is libelf. I had this issue in the past when libelf is spack installed.

libomptarget CMake found libelf library files but didn’t check the header files.
This caused all the complaints about <libelf/libelf.h> not found.
The workaround in my case is apt instal libelf-dev.

I think the libelf search needs to be more robust in libomptarget.

Best,

Ye

Jon,

With this CMake configuration, build fails on my X86 environment.

-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    COMPILER_RT_BUILD_BUILTINS
    LLVM_CONFIG_PATH
    LLVM_DEFAULT_TARGET_TRIPLE
    LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
    LLVM_ENABLE_PROJECTS_USED
    LLVM_HOST_TRIPLE

-- Build files have been written to: /tmp/llvm/runtimes/runtimes-bins
[451/454] Performing build step for 'runtimes'
[142/173] Linking CXX shared library openmp/libomptarget/libomptarget.so
FAILED: openmp/libomptarget/libomptarget.so
: && /tmp/llvm/./bin/clang++ --target=x86_64-unknown-linux-gnu -fPIC
-fPIC -fvisibility-inlines-hidden -Werror=date-time
-Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter
-Wwrite-strings -Wcast-qual -Wmissing-field-initializers
-Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type
-Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override
-Wno-comment -Wstring-conversion -Wmisleading-indentation
-fdiagnostics-color -ffunction-sections -fdata-sections -Wall
-Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare
-Wno-extra -Wno-pedantic -std=gnu++14 -O3 -DNDEBUG -Wl,-z,defs
-Wl,-z,nodelete -shared -Wl,-soname,libomptarget.so -o
openmp/libomptarget/libomptarget.so
openmp/libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o
/tmp/llvm/lib/libLLVMSupport.a -ldl
-Wl,--version-script=/p/scratch/cjzam11/kitayama1/projects/llvm-project/openmp/libomptarget/src/exports
-lrt -ldl -lpthread -lm
/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0/lib/libz.so
/p/software/jurecadc/stages/2020/software/ncurses/6.2-GCCcore-9.3.0/lib/libtinfo.so
/tmp/llvm/lib/libLLVMDemangle.a && :
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<(anonymous namespace)::RealFSDirIter,
std::allocator<(anonymous namespace)::RealFSDirIter>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN12_GLOBAL__N_113RealFSDirIterESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<(anonymous
namespace)::CombiningDirIterImpl, std::allocator<(anonymous
namespace)::CombiningDirIterImpl>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN12_GLOBAL__N_120CombiningDirIterImplESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<llvm::vfs::(anonymous
namespace)::InMemoryDirIterator, std::allocator<llvm::vfs::(anonymous
namespace)::InMemoryDirIterator>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs12_GLOBAL__N_119InMemoryDirIteratorESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<RedirectingFSDirRemapIterImpl,
std::allocator<RedirectingFSDirRemapIterImpl>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceI29RedirectingFSDirRemapIterImplSaIS0_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info[_ZNSt23_Sp_counted_ptr_inplaceI29RedirectingFSDirRemapIterImplSaIS0_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info]+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<llvm::vfs::RedirectingFSDirIterImpl,
std::allocator<llvm::vfs::RedirectingFSDirIterImpl>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs24RedirectingFSDirIterImplESaIS2_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info[_ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs24RedirectingFSDirIterImplESaIS2_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info]+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o):VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs6detail15RecDirIterStateESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info[_ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs6detail15RecDirIterStateESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info]+0x13):
more undefined references to
`std::_Sp_make_shared_tag::_S_eq(std::type_info const&)' follow
clang-13: error: linker command failed with exit code 1 (use -v to see
invocation)
[172/173] Building CXX object openmp/r.../CMakeFiles/omp.dir/kmp_settings.cpp.o
ninja: build stopped: subcommand failed.
FAILED: runtimes/runtimes-stamps/runtimes-build
cd /tmp/llvm/runtimes/runtimes-bins &&
/p/software/jurecadc/stages/2020/software/CMake/3.18.0-GCCcore-9.3.0/bin/cmake
--build .
ninja: build stopped: subcommand failed.

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/tmp
-DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_80
-DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES=80 -DLLVM_ENABLE_PROJECTS="clang"
-DLLVM_ENABLE_RUNTIMES=openmp
-DLLVM_TARGETS_TO_BUILD="X86;NVPTX;AMDGPU"
$SCRATCH/kitayama1/projects/llvm-project/llvm

Reducing the targets to only two (X86 and NVPTX) did not help either.

[3338/3400] Performing configure step for 'runtimes'
[...]
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    COMPILER_RT_BUILD_BUILTINS
    LLVM_CONFIG_PATH
    LLVM_DEFAULT_TARGET_TRIPLE
    LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
    LLVM_ENABLE_PROJECTS_USED
    LLVM_HOST_TRIPLE

-- Build files have been written to: /tmp/llvm/runtimes/runtimes-bins
[3397/3400] Performing build step for 'runtimes'
[142/173] Linking CXX shared library openmp/libomptarget/libomptarget.so
FAILED: openmp/libomptarget/libomptarget.so
: && /tmp/llvm/./bin/clang++ --target=x86_64-unknown-linux-gnu -fPIC
-fPIC -fvisibility-inlines-hidden -Werror=date-time
-Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter
-Wwrite-strings -Wcast-qual -Wmissing-field-initializers
-Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type
-Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override
-Wno-comment -Wstring-conversion -Wmisleading-indentation
-fdiagnostics-color -ffunction-sections -fdata-sections -Wall
-Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare
-Wno-extra -Wno-pedantic -std=gnu++14 -O3 -DNDEBUG -Wl,-z,defs
-Wl,-z,nodelete -shared -Wl,-soname,libomptarget.so -o
openmp/libomptarget/libomptarget.so
openmp/libomptarget/src/CMakeFiles/omptarget.dir/api.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/device.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/interface.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/rtl.cpp.o
openmp/libomptarget/src/CMakeFiles/omptarget.dir/omptarget.cpp.o
/tmp/llvm/lib/libLLVMSupport.a -ldl
-Wl,--version-script=/p/scratch/cjzam11/kitayama1/projects/llvm-project/openmp/libomptarget/src/exports
-lrt -ldl -lpthread -lm
/p/software/jurecadc/stages/2020/software/zlib/1.2.11-GCCcore-9.3.0/lib/libz.so
/p/software/jurecadc/stages/2020/software/ncurses/6.2-GCCcore-9.3.0/lib/libtinfo.so
/tmp/llvm/lib/libLLVMDemangle.a && :
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<(anonymous namespace)::RealFSDirIter,
std::allocator<(anonymous namespace)::RealFSDirIter>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN12_GLOBAL__N_113RealFSDirIterESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<(anonymous
namespace)::CombiningDirIterImpl, std::allocator<(anonymous
namespace)::CombiningDirIterImpl>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN12_GLOBAL__N_120CombiningDirIterImplESaIS1_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<llvm::vfs::(anonymous
namespace)::InMemoryDirIterator, std::allocator<llvm::vfs::(anonymous
namespace)::InMemoryDirIterator>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs12_GLOBAL__N_119InMemoryDirIteratorESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<RedirectingFSDirRemapIterImpl,
std::allocator<RedirectingFSDirRemapIterImpl>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceI29RedirectingFSDirRemapIterImplSaIS0_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info[_ZNSt23_Sp_counted_ptr_inplaceI29RedirectingFSDirRemapIterImplSaIS0_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info]+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o): in function
`std::_Sp_counted_ptr_inplace<llvm::vfs::RedirectingFSDirIterImpl,
std::allocator<llvm::vfs::RedirectingFSDirIterImpl>,
(__gnu_cxx::_Lock_policy)2>::_M_get_deleter(std::type_info const&)':
VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs24RedirectingFSDirIterImplESaIS2_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info[_ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs24RedirectingFSDirIterImplESaIS2_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info]+0x13):
undefined reference to `std::_Sp_make_shared_tag::_S_eq(std::type_info
const&)'
/p/software/jurecadc/stages/2020/software/binutils/2.34-GCCcore-9.3.0/bin/ld:
/tmp/llvm/lib/libLLVMSupport.a(VirtualFileSystem.cpp.o):VirtualFileSystem.cpp:(.text._ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs6detail15RecDirIterStateESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info[_ZNSt23_Sp_counted_ptr_inplaceIN4llvm3vfs6detail15RecDirIterStateESaIS3_ELN9__gnu_cxx12_Lock_policyE2EE14_M_get_deleterERKSt9type_info]+0x13):
more undefined references to
`std::_Sp_make_shared_tag::_S_eq(std::type_info const&)' follow
clang-13: error: linker command failed with exit code 1 (use -v to see
invocation)
[172/173] Building CXX object openmp/r.../CMakeFiles/omp.dir/kmp_settings.cpp.o
ninja: build stopped: subcommand failed.
FAILED: runtimes/runtimes-stamps/runtimes-build
cd /tmp/llvm/runtimes/runtimes-bins &&
/p/software/jurecadc/stages/2020/software/CMake/3.18.0-GCCcore-9.3.0/bin/cmake
--build .
ninja: build stopped: subcommand failed.

If I build with -DLLVM_ENABLE_PROJECTS="clang;openmp" , then three
targets NVPTX, AMDGPU, and X86 were built without a failure.
Jon, I'd appreciate if you could take a look at the build with
ENABLE_RUNTIMES case closely. Thanks!