MLIR Build Error when enabling cuda runner

MLIR
1

I tried to build & install MLIR with cuda runner enabled. But I got some errors when linking.

Here are my build commands:

mkdir llvm-project/build
cd llvm-project/build
cmake -G Ninja ../llvm \
   -DLLVM_ENABLE_PROJECTS=mlir \
   -DLLVM_BUILD_EXAMPLES=ON \
   -DLLVM_BUILD_LLVM_DYLIB=ON \
   -DLLVM_INSTALL_UTILS=ON \
   -DLLVM_TARGETS_TO_BUILD="X86;NVPTX;AMDGPU" \
   -DCMAKE_BUILD_TYPE=Release \
   -DLLVM_ENABLE_ASSERTIONS=ON \
   -DCMAKE_INSTALL_PREFIX=/path/to/llvm/install \
   -DMLIR_ENABLE_CUDA_RUNNER=ON
ninja -j32 && ninja install

And I got the following error:

[1/2] Linking CXX shared library lib/libMLIR.so.16git
FAILED: lib/libMLIR.so.16git
<a number of linked files but no libcuda.so>:
tools/mlir/lib/Dialect/GPU/CMakeFiles/obj.MLIRGPUTransforms.dir/Transforms/SerializeToCubin.cpp.o: In function `emitCudaError(llvm::Twine const&, char const*, cudaError_enum, mlir::Location)':
SerializeToCubin.cpp:(.text._ZL13emitCudaErrorRKN4llvm5TwineEPKc14cudaError_enumN4mlir8LocationE+0x48): undefined reference to `cuGetErrorString'
tools/mlir/lib/Dialect/GPU/CMakeFiles/obj.MLIRGPUTransforms.dir/Transforms/SerializeToCubin.cpp.o: In function `(anonymous namespace)::SerializeToCubinPass::serializeISA(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x7b): undefined reference to `cuInit'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0xeb): undefined reference to `cuDeviceGet'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x161): undefined reference to `cuCtxCreate_v2'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x222): undefined reference to `cuLinkCreate_v2'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x324): undefined reference to `cuLinkAddData_v2'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x3a4): undefined reference to `cuLinkComplete'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x455): undefined reference to `cuLinkDestroy'
SerializeToCubin.cpp:(.text._ZN12_GLOBAL__N_120SerializeToCubinPass12serializeISAERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x4c0): undefined reference to `cuCtxDestroy_v2'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

I tried to modify CMakeLists to add libcuda.so manually but the problem remains.

I don’t know how to solve the problem, hope someone can help me, thanks!

1 Like
2

What is the output of find /usr/lib -iname 'libcuda.so*'?

3 
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.465.19.01
/usr/lib/x86_64-linux-gnu/libcuda.so.1
4

That looks as expected.

Now I see you are building a shared lib. I suspect this is a bug then. It’s probably not that it is not found but libcuda.so is set as a private dependency to the MLIRGPUTransforms library. Maybe try to change the following in lib/Dialect/GPU/CMakeLists.txt:

# find_library(CUDA_DRIVER_LIBRARY cuda)

  # target_link_libraries(MLIRGPUTransforms
  #   PRIVATE
  #   MLIRNVVMToLLVMIRTranslation
  #   ${CUDA_DRIVER_LIBRARY}
  # )

  find_library(CUDA_DRIVER_LIBRARY cuda REQUIRED)

  target_link_libraries(MLIRGPUTransforms
    PRIVATE
    MLIRNVVMToLLVMIRTranslation
  )

  target_link_libraries(MLIRGPUTransforms
    PUBLIC
    ${CUDA_DRIVER_LIBRARY}
  )
5

It doesn’t work. I change the CMakeLists.txt and build again but the problem still remains

6

I reproduced. Here’s the patch

diff --git a/mlir/tools/mlir-shlib/CMakeLists.txt b/mlir/tools/mlir-shlib/CMakeLists.txt
index 32fe833cee4e..729517e875aa 100644
--- a/mlir/tools/mlir-shlib/CMakeLists.txt
+++ b/mlir/tools/mlir-shlib/CMakeLists.txt
@@ -31,6 +31,11 @@ if(MLIR_LINK_MLIR_DYLIB)
 endif()
 
 if(LLVM_BUILD_LLVM_DYLIB)
+  if(MLIR_ENABLE_CUDA_RUNNER)
+    find_library(CUDA_DRIVER_LIBRARY cuda REQUIRED)
+    list(APPEND _DEPS ${CUDA_DRIVER_LIBRARY})    
+  endif()
+
   add_mlir_library(
     MLIR
     SHARED

This should work for you. I haven’t submitted a new patch to Phabricator because I think a more general solution for the shared library should link against the stub library stubs/libcuda.so shipped with the CUDA toolkit distribution.

Edit: ⚙ D135981 [mlir] Fix CMake build for libMLIR.so with MLIR_ENABLE_CUDA_RUNNER=ON

7

Yes this does work. Thanks for your help!

8

Hi, I’m Compiling MLIR with commands below:

cmake -G Ninja ../llvm \
   -DLLVM_ENABLE_PROJECTS=mlir \
   -DLLVM_BUILD_EXAMPLES=ON \
   -DLLVM_TARGETS_TO_BUILD="Native;NVPTX" \
   -DCMAKE_BUILD_TYPE=Release \
   -DLLVM_ENABLE_ASSERTIONS=ON \
   -DCMAKE_C_COMPILER=clang-15 \
   -DCMAKE_CXX_COMPILER=clang++-15 \
   -DLLVM_ENABLE_LLD=ON \
   -DLLVM_ENABLE_RTTI=ON \
   -DMLIR_ENABLE_CUDA_RUNNER=ON

But I don’t have mlir-gpu-runner in my ./build/bin/ folder.

The output of find /usr/lib -iname 'libcuda.so*' is:

/usr/lib/i386-linux-gnu/libcuda.so.1
/usr/lib/i386-linux-gnu/libcuda.so.525.147.05
/usr/lib/i386-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so.525.147.05
/usr/lib/x86_64-linux-gnu/libcuda.so
9

This is expected: there is no mlir-gpu-runner in the project.

10

Ou, well, I thought there were programs like mlir-cpu-runner. :smiley: