The libc implementation for the GPUs

[ikitayama]

@jdoerfert et al., I suppose the proposed libc implementation will be written to the LibC.{h,cpp} files, am I correct?

[ikitayama]

Currently only a few pieces of code there though.

[tschuett]

Are there plans to cooperate with the LLVM libc?

[jdoerfert]

The design of GPU libc is not decided yet. (let’s not jump to libc++ yet as it is even less clear and libm might also be a different story depending on what we do).

There are three options, and ongoing development (RPC support) that plays into this. Options:

1. Implement our own device subset of libc, in LibC.{h,cpp}, and use RPC for functions we want to support via the host.
2. Port a simple libc onto the device, use RPC for the hard parts or disable them for now. Simple libc implementations exist out there.
3. Retarget llvm/libc for GPUs, RPC again for host parts we want to support.

The complexity grows as you move down the list but also the capabilities and code reuse. We are going to evaluate option 3 soon and report back. If people are interested in this, let me know.

An early RPC prototype, and some more libc function support (via option 1), were explored as part of this work recently: https://tianshilei.me/wp-content/uploads/2022/09/llvm-hpc-22.pdf
That said, we will not continue with exactly this RPC impl. as it requires manual stubs and writing libc functions (via option 1) turned out to be tedious…

[tschuett]

During the mdspan discussion:

Contributing MDSpan implementation

Fair enough. Probably defer that for now, but I will consider writing that RFC at least for the parts of libc++ which are header only inlineable code without dynamic memory allocations - i.e. the pieces where its literally just function markup without any other implications for the underlying implementation (math functions, numeric traits, tuple, array, etc.). But I guess we should start with building some credibility with providing a good pure standard C++ mdspan implementation first :-D.

A similar topic came up: libc++ for GPUs.

[jdoerfert]

I saw that, but I am unsure what this is supposed to mean. If you want the header only parts that do not pull in lib(std)c++ libraries, you can already use it. This is also unrelated of dynamic memory allocations or “inlineable”.

[tschuett]

IIRC, LLVM libc is very modular. If you want to start with strlen and hate printf, you can create your own config.

llvm-project/libc/config at main · llvm/llvm-project

[ikitayama]

@tschuett do they have a timeframe as to when the libc will have all the functions that GNU libc?

[tschuett]

Unfortunately, no. I am not a libc developer. They made a lot of progress. And as I hinted above it is a modular design. You pick strlen today and in a month you can add memcpy.

[ikitayama]

I’m interested in testing the option 3 prototype.

[tschuett]

On the top left is a status category:
https://libc.llvm.org/index.html

[tschuett]

There will be a porting guide for new targets, i.e., NVIDIA_GPU_SM30, soonish.
https://reviews.llvm.org/D136960

[jdoerfert]

Great. Will make it easier to try out a GPU port! Once we give it a shot we’ll comment here again.

[ikitayama]

The order is libc first, then libc++ follows after that?

[jhuber6]

The first support forlibc functions was landed upstream with documentation here. Right now it only supports the string.h and ctype.h functions. We are currently developing an RPC mechanism to enable calling functions that require OS support. Feedback would be appreciated.

[ikitayama]

@jhuber6 I am trying to test it though at build time I get:

-- Configuring incomplete, errors occurred!
See also "/home/ubuntu/projects/llvm-project/build/runtimes/runtimes-bins/CMakeFiles/CMakeOutput.log".
See also "/home/ubuntu/projects/llvm-project/build/runtimes/runtimes-bins/CMakeFiles/CMakeError.log".
FAILED: runtimes/runtimes-stamps/runtimes-configure /home/ubuntu/projects/llvm-project/build/runtimes/runtimes-stamps/runtimes-configure
cd /home/ubuntu/projects/llvm-project/build/runtimes/runtimes-bins && /usr/bin/cmake -DCMAKE_C_COMPILER=/home/ubuntu/projects/llvm-project/build/./bin/clang -DCMAKE_CXX_COMPILER=/home/ubuntu/projects/llvm-project/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/home/ubuntu/projects/llvm-project/build/./bin/clang -DCMAKE_LINKER=/home/ubuntu/projects/llvm-project/build/./bin/ld.lld -DCMAKE_AR=/home/ubuntu/projects/llvm-project/build/./bin/llvm-ar -DCMAKE_RANLIB=/home/ubuntu/projects/llvm-project/build/./bin/llvm-ranlib -DCMAKE_NM=/home/ubuntu/projects/llvm-project/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/home/ubuntu/projects/llvm-project/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/home/ubuntu/projects/llvm-project/build/./bin/llvm-objcopy -DCMAKE_STRIP=/home/ubuntu/projects/llvm-project/build/./bin/llvm-strip -DCMAKE_READELF=/home/ubuntu/projects/llvm-project/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/home/ubuntu/opt/llvm -DLLVM_BINARY_DIR=/home/ubuntu/projects/llvm-project/build -DLLVM_CONFIG_PATH=/home/ubuntu/projects/llvm-project/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF -DLLVM_LIT_ARGS=-sv -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=16.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DCMAKE_C_COMPILER_LAUNCHER= -DCMAKE_CXX_COMPILER_LAUNCHER= -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_TARGET_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON "-DLLVM_ENABLE_RUNTIMES=libc;openmp" -DLLVM_LIBCXX_USED=0 -DLLVM_LIBC_FULL_BUILD=ON -GNinja /home/ubuntu/projects/llvm-project/llvm/runtimes/../../runtimes && /usr/bin/cmake -E touch /home/ubuntu/projects/llvm-project/build/runtimes/runtimes-stamps/runtimes-configure
ninja: build stopped: subcommand failed.```

[ikitayama]

This is how I invoked cmake:

cmake -B build -S ./llvm -G Ninja \
   -DLLVM_ENABLE_PROJECTS="clang;lld;compiler-rt" \
   -DLLVM_ENABLE_RUNTIMES="libc;openmp" \
   -DCMAKE_BUILD_TYPE=Release \
   -DLLVM_LIBC_FULL_BUILD=ON \
   -DCMAKE_INSTALL_PREFIX=$HOME/opt/llvm

[tschuett]

You are missing at least:

   -DLLVM_LIBC_TARGET_OS=gpu           \ # Build in GPU mode
   -DLLVM_LIBC_GPU_ARCHITECTURES=all   \ # Build all supported architectures

[ikitayama]

-- Set COMPILER_RESOURCE_DIR to /tmp/build/lib/clang/16 using --print-resource-dir
CMake Error at /home/ubuntu/projects/llvm-project/libc/cmake/modules/LLVMLibCCheckCpuFeatures.cmake:83 (message):
  Failed to compile: Change Dir:
  /tmp/build/runtimes/runtimes-bins/libc/check_/CMakeFiles/CMakeTmp



  Run Build Command(s):/usr/bin/ninja cmTC_1c65e && [1/2] Building CXX object
  CMakeFiles/cmTC_1c65e.dir/check_cpu_features.cpp.o

  FAILED: CMakeFiles/cmTC_1c65e.dir/check_cpu_features.cpp.o

  /tmp/build/./bin/clang++ --target=aarch64-unknown-linux-gnu -Wall -Wextra
  -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wdelete-non-virtual-dtor
  -Wno-comment -fdiagnostics-color -MD -MT
  CMakeFiles/cmTC_1c65e.dir/check_cpu_features.cpp.o -MF
  CMakeFiles/cmTC_1c65e.dir/check_cpu_features.cpp.o.d -o
  CMakeFiles/cmTC_1c65e.dir/check_cpu_features.cpp.o -c
  /tmp/build/runtimes/runtimes-bins/libc/cpu_features/check_cpu_features.cpp


  /tmp/build/runtimes/runtimes-bins/libc/cpu_features/check_cpu_features.cpp:1:10:
  fatal error: 'cstdio' file not found

  #include <cstdio>

           ^~~~~~~~

  1 error generated.

  ninja: build stopped: subcommand failed.



Call Stack (most recent call first):
  /home/ubuntu/projects/llvm-project/libc/CMakeLists.txt:136 (include)


-- Configuring incomplete, errors occurred!
See also "/tmp/build/runtimes/runtimes-bins/CMakeFiles/CMakeOutput.log".
See also "/tmp/build/runtimes/runtimes-bins/CMakeFiles/CMakeError.log".
FAILED: runtimes/runtimes-stamps/runtimes-configure /tmp/build/runtimes/runtimes-stamps/runtimes-configure
cd /tmp/build/runtimes/runtimes-bins && /usr/bin/cmake -DCMAKE_C_COMPILER=/tmp/build/./bin/clang -DCMAKE_CXX_COMPILER=/tmp/build/./bin/clang++ -DCMAKE_ASM_COMPILER=/tmp/build/./bin/clang -DCMAKE_LINKER=/tmp/build/./bin/ld.lld -DCMAKE_AR=/tmp/build/./bin/llvm-ar -DCMAKE_RANLIB=/tmp/build/./bin/llvm-ranlib -DCMAKE_NM=/tmp/build/./bin/llvm-nm -DCMAKE_OBJDUMP=/tmp/build/./bin/llvm-objdump -DCMAKE_OBJCOPY=/tmp/build/./bin/llvm-objcopy -DCMAKE_STRIP=/tmp/build/./bin/llvm-strip -DCMAKE_READELF=/tmp/build/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/usr/local -DLLVM_BINARY_DIR=/tmp/build -DLLVM_CONFIG_PATH=/tmp/build/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF -DLLVM_LIT_ARGS=-sv -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=16.0.0git -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=/usr/bin/ninja -DCMAKE_C_COMPILER_LAUNCHER= -DCMAKE_CXX_COMPILER_LAUNCHER= -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DLLVM_BUILD_TOOLS=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DHAVE_LLVM_LIT=ON "-DLLVM_ENABLE_RUNTIMES=libc;openmp" -DLLVM_LIBCXX_USED=0 -DLLVM_LIBC_FULL_BUILD=ON -DLLVM_LIBC_GPU_ARCHITECTURES=all -DLLVM_LIBC_TARGET_OS=gpu -GNinja /home/ubuntu/projects/llvm-project/llvm/runtimes/../../runtimes && /usr/bin/cmake -E touch /tmp/build/runtimes/runtimes-stamps/runtimes-configure
[4402/4441] Building CXX object tools/...CMakeFiles/yaml2obj.dir/yaml2obj.cpp.o
ninja: build stopped: subcommand failed.

[ikitayama]

I’ve added the variables missing.