How to compile C++ files under compiler-rt with -stdlib=libc++?

Runtimes
1

I am using the following commands to build runtimes.

$ cmake \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=<install-path> \
-DLLVM_ENABLE_ASSERTIONS=OFF \
-DCOMPILER_RT_USE_LIBCXX=ON \
-DLLVM_TARGETS_TO_BUILD=X86 \
'-DLLVM_RUNTIME_TARGETS=x86_64-unknown-linux-gnu;i386-unknown-linux-gnu' \
'-DLLVM_ENABLE_PROJECTS=clang;lld'  \
'-DLLVM_ENABLE_RUNTIMES=compiler-rt;libcxx;libcxxabi;libunwind' \
'-DCMAKE_CXX_FLAGS=-stdlib=libc++' \
'-DCMAKE_EXE_LINKER_FLAGS=-rtlib=compiler-rt' \
'-DCMAKE_SHARED_LINKER_FLAGS=-rtlib=compiler-rt' \
'-DCMAKE_MODULE_LINKER_FLAGS=-rtlib=compiler-rt' \
-DCMAKE_CXX_COMPILER=<path-to-clang++> \
-DCMAKE_C_COMPILER=<path-to-clang> \
-S <source-dir> \
-B build
$ ninja -C build runtimes

But the build fails with errors in various C++ files under compiler-rt (e.g. llvm-project/compiler-rt/lib/orc/macho_platform.cpp) because clang++ (which happens to be the clang++ that was just built) uses the system libstdc++ which is very old and does not support C++14.

You may notice that I have added -stdlib=libc++ to CMAKE_CXX_FLAGS and also tried -DCOMPILER_RT_USE_LIBCXX but they don’t help. But when I manually add -stdlib=libc++ to build.ninja, the files failing to compile go through.

What cmake configuration option can I use to make clang use libc++? The clang installation that I used to build LLVM and the new clang also contains libc++ but that is not what ends up getting used when building runtimes.

Or here is a more general question. What are the proper steps for creating a self-contained clang/libc++/libc++abi/compiler-rt/libunwind installation?

I used to be able to do this before (e.g. version 11.0.0) but the recent changes to building libcxx, compiler-rt and libunwind have broken my build scripts. I have tried to follow these instructions but they don’t work because the system libstdc++ is old and the newly built clang picks it up instead of libc++ (that was just built) when building the C++ sources under compiler-rt.

Previously, it used to work because the clang that I provide with -DCMAKE_CXX_COMPILER was used to build everything the first time around.

2

Can I get some pointers to a solution for this problem? I tried the instructions for building a distribution but run into the same problem. The clang++ built in stage1 is not able to compile compiler-rt because it is using the old system libstdc++ instead of the libc++ that it just built. How can I get -stdlib=libc++ to be passed to this clang? -DCMAKE_CXX_FLAGS=-stdlib=libc++ doesn’t help.

3

The runtimes builds do their own separate CMake configures, so variables you’ve specified in your initial CMake invocation won’t necessarily get passed down to them. The logic is in https://github.com/llvm/llvm-project/blob/main/llvm/runtimes/CMakeLists.txt, and the relevant bit for your case is https://github.com/llvm/llvm-project/blob/da047445f77bfd74b04c36169e104f35dbfff84e/llvm/runtimes/CMakeLists.txt#L232-L258. Looking at that, it seems like any variables specified in RUNTIMES_CMAKE_ARGS will be passed down to the runtimes configuration, so adding -DRUNTIMES_CMAKE_ARGS='-DCOMPILER_RT_USE_LIBCXX=ON' should do the trick. If you want the clang you’re building to default to libc++, you could pass -DCLANG_DEFAULT_CXX_STDLIB=libc++ to your initial configure instead.

@petrhosek builtin_default_target adds COMPILER_RT to its PASSTHROUGH_PREFIXES, but runtime_default_target doesn’t. Do you know if this is intentional or just an oversight?

4

Thanks. -DRUNTIMES_CMAKE_ARGS='-DCOMPILER_RT_USE_LIBCXX=ON' did not do the trick :frowning:

I am a little confused how distribution builds are supposed to work if the system C++ compiler and libstdc++ are older (in my case, it is CentOS 7 with gcc 4.8.5) and newer compilers and C++ library are in some non-standard path.

So I am doing the stage 1 build with gcc 9.2.0 which is available in a non-standard path. stage 1 clang builds fine but then when compiler-rt is built using the just built clang, it fails on C++ files like llvm-project/compiler-rt/lib/orc/macho_platform.cpp . It is spewing out errors because clang is picking up old C++ headers from the system path.

Even when I follow the instructions for building a distribution, it fails on macho_platform.cpp.

My case is not atypical (system libraries being old but having an installation of a newer compiler and libraries elsewhere). So what I am missing?

5

Hmm. There should be a file like llvm/runtimes/runtimes-bins/CMakeCache.txt in your build directory (might not be at that exact path, but searching for files named CMakeCache.txt in your build directory should find the one I’m after). Does that include the COMPILER_RT_USE_LIBCXX entry?

6

I started looking at this again…

The top-level cmake emits this warning:

CMake Warning:
  Manually-specified variables were not used by the project:

    RUNTIMES_CMAKE_ARGS

COMPILER_RT_USE_LIBCXX is defined CMakeCache.txt though:

$ grep COMPILER_RT_USE_LIBCXX llvm/runtimes/runtimes-x86_64-unknown-linux-gnu-bins/CMakeCache.txt
COMPILER_RT_USE_LIBCXX:BOOL=ON

But compiler-rt still fails to build:

/u/riyaz/src/llvm-project/compiler-rt/lib/orc/error.h:234:12: error: no template named 'enable_if_t' in namespace 'std'; did you mean 'enable_if'?
      std::enable_if_t<!std::is_convertible<OtherT, T>::value> * = nullptr) {
      ~~~~~^~~~~~~~~~~
           enable_if
/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../include/c++/4.8.5/type_traits:1766:12: note: 'enable_if' declared here
    struct enable_if 
           ^
In file included from /u/riyaz/src/llvm-project/compiler-rt/lib/orc/macho_platform.cpp:13:
In file included from /u/riyaz/src/llvm-project/compiler-rt/lib/orc/macho_platform.h:17:
In file included from /u/riyaz/src/llvm-project/compiler-rt/lib/orc/executor_address.h:20:
In file included from /u/riyaz/src/llvm-project/compiler-rt/lib/orc/simple_packed_serialization.h:39:
/u/riyaz/src/llvm-project/compiler-rt/lib/orc/error.h:372:10: error: no template named 'aligned_union_t' in namespace 'std'
    std::aligned_union_t<1, storage_type> TStorage;
    ~~~~~^
/u/riyaz/src/llvm-project/compiler-rt/lib/orc/error.h:373:10: error: no template named 'aligned_union_t' in namespace 'std'
    std::aligned_union_t<1, error_type> ErrorStorage;
    ~~~~~^

Why is it picking up the system libstdc++ from /usr/lib/gcc/..../4.8.5/.... even though COMPILER_RT_USE_LIBCXX=ON is specified and there is libc++.* under build/lib/x86_64-unknown-linux-gnu/?

7

Are you supposed to use llvm-project/llvm/runtimes as the source dir? When I switched some builds over to “the default build” I used the top-level llvm-project/runtimes and I recall that I got different results if I accidentally used llvm-project/llvm/runtimes.

This might be entirely unrelated to your issues reported here, though.

8

Hi,
I think I have a related problem. I did not the full picture yet, but so far my best guess is, that is somehow like this:

  • While compiling with gcc the includes of the gcc are taken automatically (by gcc) and therefore everything is properly defined.
  • For runtimes clang is used. Because the include-path of the gcc includes are not given to CFLAGS/CXXFLAGS (or how it has to be defined) the definitions are not available yet and because everything is compiled together clang is not installed so far and the includes for clang does not exist yet. Therefore only the system include files are seen.

At least I assume in my case something like this. One solution could be to provide the gcc includes for clang, but I think that is not clean. Another way could be to first install clang and then compile the runtimes with a proper installed clang. Not sure if that works out.

Maybe I am also totally wrong, but that is so far my best guess.

9

the system C++ compiler and libstdc++ are older […] and newer compilers and C++ library are in some non-standard path.

This is my situation and I’m running into the same issues (libstdc++ getting picked up, etc).

@RVP Did you ever discover a solution here?

10

Sorry for the late response (I just saw this when I started looking at this again to upgrade my build). I had worked around this issue by manually modifying CMakeCache.txt file. I do know why the issue occurs but I don’t know if it is an oversight in llvm/runtimes/CMakeLists.txt. The cmake configuration for runtimes build discards most of the cmake options provided to the top-level cmake. This was pointed out by @smeenai above.

@smeenai suggested using RUNTIMES_CMAKE_ARGS but that doesn’t work because the functions runtime_default_target and runtime_register_target call llvm_External_Project_Add differently. The former passes RUNTIMES_CMAKE_ARGS but the latter doesn’t. So if you add LLVM_ENABLE_RUNTIME_TARGETS, runtime_register_target gets used instead of runtime_default_target.

I don’t whether not passing RUNTIMES_CMAKE_ARGS down (when LLVM_ENABLE_RUNTIME_TARGETS is specified) is intentional or a bug. Even making a clang++ that defaults to libc++, compiler-rt and libunwind doesn’t help, because when variables like …_USE_COMPILER_RT are not passed down to the runtimes build cmake, gcc_s, libstdc++, etc. get added to the link flags. :frowning: (regardless of clang’s default).

11

To pass variables to runtimes you need to do:

-DRUNTIMES_<target>_COMPILER_RT_BLAH...

1 Like
12

I don’t see RUNTIME_<target>_COMPILER_RT.... here:

# runtime_register_target(target)
#   Utility function to register external runtime target.
function(runtime_register_target name target)
:
:
  llvm_ExternalProject_Add(runtimes-${name}
                           ${CMAKE_CURRENT_SOURCE_DIR}/../../runtimes
                           DEPENDS ${${name}_deps}
                           # Builtins were built separately above
                           CMAKE_ARGS -DCOMPILER_RT_BUILD_BUILTINS=Off
                                      -DLLVM_INCLUDE_TESTS=${LLVM_INCLUDE_TESTS}
                                      -DLLVM_DEFAULT_TARGET_TRIPLE=${target}
                                      -DLLVM_ENABLE_PROJECTS_USED=${LLVM_ENABLE_PROJECTS_USED}
                                      -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR}
                                      -DCMAKE_C_COMPILER_WORKS=ON
                                      -DCMAKE_CXX_COMPILER_WORKS=ON
                                      -DCMAKE_ASM_COMPILER_WORKS=ON
                                      -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON
                                      -DLLVM_RUNTIMES_TARGET=${name}
                                      ${COMMON_CMAKE_ARGS}
                                      ${${name}_extra_args}
                           EXTRA_TARGETS ${${name}_extra_targets}
                                         ${${name}_test_targets}
                           USE_TOOLCHAIN
                           TARGET_TRIPLE ${target}
                           ${EXTRA_ARGS})

Where is this passed to cmake for runtimes? The variables that need to be passed through to cmake for runtimes are:

LIBUNWIND_USE_COMPILER_RT
LIBCXXABI_USE_COMPILER_RT
LIBCXXABI_USE_LLVM_UNWINDER
LIBCXX_USE_COMPILER_RT

Do you mean that for each target (in my case: -DLLVM_RUNTIME_TARGETS=x86_64-unknown-linux-gnu;i386-unknown-linux-gnu), I have to pass:

 -DRUNTIMES_<target>_LIBUNWIND_USE_COMPILER_RT=On
 -DRUNTIMES_<target>_LIBCXXABI_USE_COMPILER_RT=On
 -DRUNTIMES_<target>_LIBCXXABI_USE_LLVM_UNWINDER=On
 -DRUNTIMES_<target>_LIBCXX_USE_COMPILER_RT=On

where <target> is x86_64-unknown-linux-gnu and i386-unknown-linux-gnu. This is very cumbersome. Even then I am not sure if it will work because it is not clear how the above variables are passed through to the runtimes cmake. There was a time (before llvm-10, I think), this was all much simpler.

The requirement is pretty simple. I want to make a clang + compiler-rt + libunwind + liccxx build that supports both x86_64 and i386 and it should have no dependency on libgcc_s or libstdc++. I would also like to make a static build (I have succeeded in doing that but with a lot of hackery). If you look at the documentation, or try to use any of the existing cache files, etc. they don’t work out of the box where I want to make a 64-bit clang compiler + 32/64 bit libraries on x86. I still have to do a lot of additional flag twiddling.

This link is useful but it works only if I make only a 64-bit build. If I want i386 libraries as well, then it doesn’t work (runs into the issue discussed in this thread).

Basically, there isn’t a simple answer to what I think should be a pretty common requirement. At least. I haven’t figured out how to do it in a simple way and I have asked for help in these forums and I have not gotten a simple answer (I suspect such an answer doesn’t exist). I am not a cmake expert by any means, and it doesn’t help that the cmake files for runtimes are a mess. I guess it has gotten to a point that the whole thing is so fragile that simplifying it without breaking a lot of existing builds isn’t easy.

13

Build system is complex, not sure there is much that can be done about that considering the different requirements.

But I usually find the Fuchsia caches the most helpful when dealing with runtimes, maybe check what they do here: llvm-project/Fuchsia-stage2.cmake at main · llvm/llvm-project · GitHub

14

The logic for including the RUNTIMES_${target}_... args in the configure is at llvm-project/CMakeLists.txt at 0de988c98c27d2d020dcee2c4dabc52e52fda382 · llvm/llvm-project · GitHub. It’s per-target because it’s very common to want to pass different flags for building runtimes for different targets, e.g. Windows vs. Linux. It’s less ideal for the case of building for the same OS but different architectures though.

What I’d recommend is using a cache file instead of putting everything in the CMake invocation. Tobias linked an example above, and you can include that in your configure with the -C flag. The cache file is just a regular CMake script, so you can e.g. have a loop to avoid duplication:

foreach(target i386-unknown-linux-gnu x86_64-unknown-linux-gnu)
  set(RUNTIMES_${target}_LIBUNWIND_USE_COMPILER_RT ON CACHE BOOL "")
endforeach()
1 Like