How to build libc++, libunwind independent of libgcc_s.so (use compiler-rt builtins, atomics)?

Runtimes
1

I used the following commands:

cmake -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -S ${LLVM_SRC} \
  -B llvm-build \
  -DCMAKE_CXX_COMPILER=${GCC_TOOLCHAIN}/bin/g++ \
  -DCMAKE_C_COMPILER=${GCC_TOOLCHAIN}/bin/gcc \
  -DLLVM_ENABLE_PROJECTS="clang" \
  -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi;libunwind" \
  -DLLVM_RUNTIME_TARGETS="x86_64-unknown-linux-gnu"

ninja -C llvm-build runtimes

But libc++abi.so.1, libc++.so.1. and libunwind.so.1 have a dependency on libgcc_s.so.1 (as reported by ldd) and libc++.so.1 has a dependency on libatomic.so.1 as well.

I was able to do this in LLVM 10 (not exactly the same cmake as above, but with all the changes that have gone in how runtimes are built, what I used to do will not work anymore, various errors that I will not go into; I had raised those issues in other threads and was asked to use the bootstrapping method above). How does one make a bootstrapping build of runtime that uses compiler-rt and not libgcc and libatomic from the GCC toolchain?

I have a related question. libc++ is compiled using a newly built clang. builtins are built when compiler-rt is listed in LLVM_ENABLE_RUNTIMES. But some components of compiler-rt contain C++ files that require libc++. So how is this cycle handled in the build?

NOTE: I have also tried using a previously built clang 10 instead ${GCC_TOOLCHAIN}/bin/g++ as the host compiler but I get the same results. The newly built clang (stage 1) does not use compiler-rt to build the runtimes and I couldn’t figure out a way to force it.

NOTE: Using llvm release/14.x

1 Like
2

Any response? Is there a compile option that will avoid linking libgcc_s with libc++, etc.?

3

You should be able to set LIBCXXABI_USE_LLVM_UNWINDER to link against libunwind and LIBCXX_USE_COMPILER_RT to link against compiler-rt.

4

libgcc or its aliases are drawn into the build of libc++ and libc++abi by one line in libunwind -
libunwind/src/AddressSpacce.hpp Line: 624

if (dladdr((void *)addr, &dyldInfo)) {

And as you said, the problem is difficult to solve because there is a cyclic dependency between compiler-rt and libcxx.

  • To build libcxx with compiler-rt (libclang_rt.builtins) you need compiler-rt to be already built.
  • Some parts of compiler-rt requires a standard C++ compiler.

There are many teams using compiler-rt instaed of libgcc that have produced different hacks to aviod this:

  • Google Android
  • Google Fuchsia
  • Google Chrome
  • Microsoft Win32
  • IBM AIX
  • Apple Darwin

Google Android’s internal flag _LIBUNWIND_USE_DLADDR is the simplest. It
avoids this call (temporarily reducing the functionality of libunwind)

Like you would argue that this is reasonable and generic requirement even in an <aarch>-<vendor>-linux-gnu environment. So, even though I am opposed to the creation of cmake flags, exposing this with a flag called something like:

  • LIBUNWIND_USE_RUNTIME=“none” (as opposed to “libgcc” or “compiler-rt”)
  • LIBUNWIND_NO_RUNTIME=ON

Would benefit us all. I (locally) patched AddressSpacce.hpp, adding

#define _LIBUNWIND_USE_DLADDR 0

at the top to remove the dependency on libgcc (your immediate hurdle.)

Your next hurdle is to rebuild libcxx, libcxxabi, libunwind using your new compiler-rt (and the unpatched AddressSpacce.hpp code) to regain the lost functionality.
I can only give you a few pointers…

  • build libunwind using compiler-rt with LIBUNWIND_USE_COMPILER_RT=ON
  • build libcxxabi using compiler-rt with LIBCXXABI_USE_COMPILER_RT=ON
  • build libcxxabi using libunwind with DLIBCXXABI_USE_LLVM_UNWINDER=ON
  • build libcxx using compiler-rt with LIBCXX_USE_COMPILER_RT=ON
  • build libcxx using libcxxabi with LIBCXX_CXX_ABI=ON

er… there is no LIBCXX_USE_LLVM_UNWINDER so

  • build libcxxabi using a static libunwind library with LIBCXXABI_ENABLE_STATIC_UNWINDER
  • build libcxx using a static libcxxabi library with LIBCXX_ENABLE_STATIC_ABI_LIBRARY unless you are not using a standard c++ abi

(these last two have been replaced in version 15 with ‘LIBCXX_CXX_ABI=libcxxabi’)

NOTE: this still leaves the possibility that libgcc code still lurks within the binaries…

The alternative to cutting down the functionality of libunwind that I am currently experimenting with is to cut down the functionality of compiler-rt so it does not need libcxx, then build it first. This requires a whole host of flags:

  • COMPILER_RT_BUILD_SANITIZERS=OFF
  • COMPILER_RT_BUILD_MEMPROF=OFF
  • COMPILER_RT_BUILD_XRAY=OFF
  • COMPILER_RT_BUILD_LIBFUZZER=OFF
  • COMPILER_RT_BUILD_ORC=OFF
  • COMPILER_RT_INCLUDE_TESTS=OFF

OK, onto libatomic!
In libcxx/cmake/config-ix.cmake line: 128 the code checks for the libatomic library even though we might be using compiler-rt.

check_library_exists(atomic __atomic_fetch_add_8 “” LIBCXX_HAS_ATOMIC_LIB)

Removing this line allows the build to complete without libatomic. I have not yet investigated any unintended consequences of this.

1 Like
5

Thank you for the detailed answer. It gives me some hints on how to proceed. I didn’t understand the following, though.

Why would dladdr require libgcc_s? Isn’t the definition for dladdr provided by libdl.so?

6

compiler-rt no longer builds atomic routines into builtins by default. See ⚙ D47606 [compiler-rt] [builtins] Don't build __atomic_load etc. by default..

7

I’ve done this recently, see my cmake cache files:
https://github.com/ClangBuiltLinux/containers/tree/main/llvm-project

Specifically, libunwind: https://github.com/ClangBuiltLinux/containers/blob/main/llvm-project/stage1.cmake#L80-L81

libc++: https://github.com/ClangBuiltLinux/containers/blob/main/llvm-project/stage1.cmake#L101-L105

8

You can build just compiler-rt builtins (and crt if needed) by using compiler-rt/lib/builtins (and compiler-rt/lib/crt respectively) as the CMake source directory. This is what the LLVM bootstrapping build does, see https://github.com/llvm/llvm-project/blob/5d135041c58f0ef87d266ae239a1fb17b70fc051/llvm/runtimes/CMakeLists.txt#L101.

If possible, I’d recommend using the bootstrapping build directly as the official recommended solution since it already automates all of these steps and is actively maintained.