Trying to build clang and runtimes but linking fails, need help

I have been unable to build clang with cross-compile runtimes for ARM64 due to linker problems. If I let it use gnu ld (2.34), the link fails because the 18GiB RAM of my Ubuntu 20.04 VM is not enough. If I configure to use lld, linking runtimes dies saying “-fuse-ld=lld” is not an option of the host compiler.

Building llvm-project 16.0.5. CMake 3.16.3. GNU compiler tools 9.4.0.

Any ideas what I’m doing wrong?

My configuration is this:

cmake -S llvm -B _build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_PARALLEL_LINK_JOBS=1 \
    -DLLVM_USE_LINKER=lld \
    -DLLVM_TARGETS_TO_BUILD="AArch64;X86" \
    -DLLVM_RUNTIME_TARGETS="AArch64;X86" \
    -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;lldb;polly" \
    -DLLVM_ENABLE_RUNTIMES="all"

The build result I get is this:

...
[5555/5568] Performing configure step for 'runtimes-X86'
CMake Warning at CMakeLists.txt:4 (message):
  Your CMake version is 3.16.3.  Starting with LLVM 17.0.0, the minimum
  version of CMake required to build LLVM will become 3.20.0, and using an
  older CMake will become an error.  Please upgrade your CMake to at least
  3.20.0 now to avoid issues in the future!


-- The C compiler identification is Clang 16.0.5
-- The CXX compiler identification is Clang 16.0.5
-- The ASM compiler identification is Clang
-- Found assembler: /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE) 
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.10") 
-- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE) 
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
-- Performing Test LLVM_RUNTIMES_LINKING_WORKS
-- Performing Test LLVM_RUNTIMES_LINKING_WORKS - Failed
-- Performing Test CXX_SUPPORTS_UNWINDLIB_EQ_NONE_FLAG
-- Performing Test CXX_SUPPORTS_UNWINDLIB_EQ_NONE_FLAG - Failed
-- Performing Test CXX_SUPPORTS_NOSTDLIBXX_FLAG
-- Performing Test CXX_SUPPORTS_NOSTDLIBXX_FLAG - Failed
-- Performing Test CXX_SUPPORTS_NOSTDINCXX_FLAG
-- Performing Test CXX_SUPPORTS_NOSTDINCXX_FLAG - Failed
-- Linker detection: LLD
-- Performing Test CXX_SUPPORTS_CUSTOM_LINKER
-- Performing Test CXX_SUPPORTS_CUSTOM_LINKER - Failed
CMake Error at /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/llvm/cmake/modules/HandleLLVMOptions.cmake:320 (message):
  Host compiler does not support '-fuse-ld=lld'
Call Stack (most recent call first):
  CMakeLists.txt:157 (include)


-- Configuring incomplete, errors occurred!
See also "/home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/runtimes/runtimes-X86-bins/CMakeFiles/CMakeOutput.log".
See also "/home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/runtimes/runtimes-X86-bins/CMakeFiles/CMakeError.log".
[5559/5568] Creating library symlink lib/liblldbIntelFeatures.so
FAILED: runtimes/runtimes-X86-stamps/runtimes-X86-configure 
CMake Error at /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/llvm/cmake/modules/HandleLLVMOptions.cmake:320 (message):
  Host compiler does not support '-fuse-ld=lld'
Call Stack (most recent call first):
  CMakeLists.txt:157 (include)

This can fail either because the compiler doesn’t know about the option, or when CMake tried to compile an example with that option, doing so failed. Most commonly because lld is not installed, or it is not available as ld.lld.

If you have apt or equivalent, sudo apt install lld should get you that. Any recent version of it should be fine.

The test that CMake does is something like <compiler> test.c -fuse-ld=lld -o /dev/null. If you can do that then CMake should be ok with it.

Though 18GiB of RAM seems very high for a single link job, but perhaps you have hit some corner case in ld in which case you’re doing the correct thing of trying lld instead.

And just for background, this is a check we wrote in llvm not a built in CMake thing, it can be found here: https://github.com/llvm/llvm-project/blob/89053e49590302c0a704d1af46ee263a5c882df6/llvm/cmake/modules/HandleLLVMOptions.cmake#L325

Pretty much try to build hello world while using lld as the linker.

Thanks @DavidSpickett .

I appear to be set up right. I can compile using -fuse-ld=lld from the command line using g++ and clang++ 16.0.5 (an earlier installation that does not have the runtimes I want). So, I am still clueless why the complaint. It is specifically when building the runtimes with the newly built compiler that I get this error.

Any other thoughts?

Right I missed that. In that case you built lld so I’m surprised it didn’t just use that. That’s how our prebuilt llvm packages work, the user doesn’t have to apt install lld seperately.

At this point I’d try to find the command line it’s trying, which means debugging the test which apparently prints to a log file: [CMake] Debugging check_cxx_source_compiles

CMake should find the right one, but double check exactly which clang 16.0.5 it’s finding. Ideally remove the old install and reset your PATH and such.

You could also confirm that there is an lld in _build/bin just in case. Sometimes I configure with many components and only build one, then forget and try to use an old copy of one of the others.

No idea if this helps but looking for “lld” in https://llvm.org/docs/CMake.html:

LLVM_ENABLE_LLD:BOOL
This option is equivalent to -DLLVM_USE_LINKER=lld, except during a 2-stage build where a dependency is added from the first stage to the second ensuring that lld is built before stage2 begins.

LLVM_RUNTIME_TARGETS option takes target triple as a value so in your case the subbuild is trying to use X86 as a triple and failing.

The -fuse-ld=lld check failing is usually a red herring, in your log output you can see that all other checks prior to it are failing as well, the linker check just happens to be the first one that terminates the CMake execution. I’d check runtimes/runtimes-X86-bins/CMakeFiles/CMakeError.log to get a better understanding of what’s going on, but it’s likely that the compiler cannot find a usable sysroot for the X86 target.

If you want to see a more complete example of how to use the boostrapping build (formerly known as the runtimes build), I’d check https://github.com/llvm/llvm-project/blob/0c46a9189c8bb559926c3391097b5c980d006f8e/clang/cmake/caches/Fuchsia-stage2.cmake

@petrhosek I see what you mean about the error message being a red herring.

I revised the config to this:

cmake -S llvm -B _build -G Ninja \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_PARALLEL_LINK_JOBS=1 \
    -DLLVM_USE_LINKER=lld \
    -DLLVM_TARGETS_TO_BUILD="AArch64;X86" \
    -DLLVM_RUNTIME_TARGETS="aarch64-unknown-linux-gnu;x86_64-unknown-linux-gnu" \
    -DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld;lldb;polly" \
    -DLLVM_ENABLE_RUNTIMES="all"

And got this error in the logs, prompting the red herring complaint about lld:

Performing C++ SOURCE FILE Test CXX_SUPPORTS_CUSTOM_LINKER failed with the following output:
Change Dir: /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/runtimes/runtimes-aarch64-unknown-linux-gnu-bins/CMakeFiles/CMakeTmp

Run Build Command(s):/usr/bin/ninja cmTC_4aaa3 && [1/2] Building CXX object CMakeFiles/cmTC_4aaa3.dir/src.cxx.o
[2/2] Linking CXX executable cmTC_4aaa3
FAILED: cmTC_4aaa3 
: && /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/./bin/clang++ --target=aarch64-unknown-linux-gnu  -DCXX_SUPPORTS_CUSTOM_LINKER  -fuse-ld=lld CMakeFiles/cmTC_4aaa3.dir/src.cxx.o  -o cmTC_4aaa3   && :
ld.lld: error: cannot open Scrt1.o: No such file or directory
ld.lld: error: cannot open crti.o: No such file or directory
ld.lld: error: cannot open crtbeginS.o: No such file or directory
ld.lld: error: unable to find library -lstdc++
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lgcc_s
ld.lld: error: unable to find library -lgcc
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lgcc_s
ld.lld: error: unable to find library -lgcc
ld.lld: error: cannot open crtendS.o: No such file or directory
ld.lld: error: cannot open crtn.o: No such file or directory
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.


Source file was:
int main() { return 0; }

So what am I still not doing right to build the Aarch64 cross-compile target? I’m not 100% sure I got the target triple right for it.

If you don’t have a standard library available you need to tell CMake to skip those steps via CMAKE_C(XX)_COMPILER_WORKS for the runtimes parts of the build, though I thought we did that automatically in the -DLLVM_ENABLE_RUNTIMES case.

I tried setting -DCMAKE_C_COMPILER_WORKS=1 and -DCMAKE_CXX_COMPILER_WORKS=1, but it didn’t change anything about the failure.

Am I leaving something out in my configuration that would resolve the standard library not being found?

I’ve read everything I could find, but I’m frustrated that there aren’t clear instructions for how to build clang so that it can cross-compile for arm64 from x86_64, complete with runtimes including santizers (I will specifically need Asan). I’m curious how anyone gets it working.

To build a cross-compiler, you want to specify -DLLVM_DEFAULT_TARGET_TRIPLE=your-target-triple or you’ll end up with a target=host non-cross compiler. It will still be able to cross-compile, but it won’t do so by default.

I don’t know whether that will help with your runtimes/sanitizer problems, I don’t ever build those.

Ok, this is specifically because you use -DLLVM_USE_LINKER=lld (or -DLLVM_ENABLE_LLD, which sets LLVM_USE_LINKER=lld). The runtimes build passes this through, but the check for the variable is a check_cxx_source_compiles, which needs a full working C++ sysroot that’s clearly not going to be present in the cross-compiling case. We just need an analogue to force-setting CMAKE_CXX_COMPILER_WORKS. Currently testing exposing an LLVM_LINKER_WORKS.

Fixing that gets further but things are still heavily broken because we’re configuring things before we’ve built other projects and checking properties by linking. In order:

  1. LLVM_USE_LINKER gets passed through (in two places) and doesn’t work when set, so needs LLVM_LINKER_WORKS
  2. libunwind checks for -fno-exceptions outside of set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) and fails again because there’s no libc++, among other things
  3. HandleLLVMOptions warns that -fPIC isn’t supported because it tries to link (both C and C++)
  4. libunwind spews warnings and errors during its build, thinking intptr_t is a plain int not a long and failing to find asm/unistd.h

Clearly the runtimes build expects to find a working set of C headers and libc for that OS (point 4), but 1-3 have nothing to do with that and require you to already have a libc++. 1 is easy to fix. 2 sounds like an ordering problem. 3 needs a bit more thought on whether it’s safe to make that just check compile rather than compile+link. 4 may or may not go away with working OS headers. Who knows what comes next. But it’s clear there’s a lot of brokenness here if you want to use -DLLVM_ENABLE_RUNTIMES for cross-compiling and that you instead need to build the projects one by one in the right order with the right flags yourself, which somewhat defeats the point.

We use the bootstrapping build to build runtimes for the following targets:

  • aarch64-unknown-fuchsia
  • aarch64-unknown-linux-gnu
  • armv7-unknown-linux-gnueabihf
  • i386-unknown-linux-gnu
  • riscv64-unknown-fuchsia
  • x86_64-unknown-linux-gnu
  • x86_64-unknown-fuchsia
  • x86_64-pc-windows-msvc

https://github.com/llvm/llvm-project/blob/0c46a9189c8bb559926c3391097b5c980d006f8e/clang/cmake/caches/Fuchsia-stage2.cmake is the cache file we use. You can see our Clang toolchain builders and if you click through you’ll eventually see the full CMake invocation. If the bootstrapping build was broken, we would have noticed.

The difference is that we pass additional flags and use different defaults, for example we pass explicit sysroot, we use LLD as a linker, compiler-rt builtins as compiler runtimes, libunwind as the unwinder and configure each runtime build as such by setting the RUNTIMES_${target}_${option} variables.

I also build various other configurations locally during development, but the full matrix of possible configurations is too large for me to test manually so it’s entirely possible that there are some configurations that don’t work and if that’s the case, we should fix them.

Unless you use an explicit sysroot, the behavior may also depend on your host installation. For example if you don’t have the aarch64-unknown-linux-gnu libraries installed on your host and you set -DLLVM_RUNTIME_TARGETS=aarch64-unknown-linux-gnu, I’d expect things are to break which may actually be the issue here (that’s why LLD is failing to find libraries such as libgcc, libc or libstdc++).

In order to better assist, it’d be helpful to explain what you’re trying to achieve, and I can try and suggest the CMake flags you might need.

I made a lot of headway by installing the following in my Ubuntu 20.04 VM:
g+±aarch64-linux-gnu
gcc-aarch64-linux-gnu
binutils-aarch64-linux-gnu
gcc-multilib
g+±multilib

This made thinks work well enough that my LLVM/clang build completes without error.

I ran “apt install clang” as well, to shake things down with the dependencies using what should be a known-good clang.

I can cross-compile to aarch64 successfully with aarch64-linux-gnu-g++, but cross-compiling with “clang --target=aarch64-linux-gnu XXX.cpp” gets a missing header error for bits/c++-config.h when including headers such as <string>.

In file included from sample.cpp:1:
/usr/bin/../lib/gcc-cross/aarch64-linux-gnu/9/../../../../include/c++/9/string:38:10: fatal error: 'bits/c++config.h' file not found
#include <bits/c++config.h>
         ^~~~~~~~~~~~~~~~~~
1 error generated.

I tried adding “–sysroot=/usr/aarch64-linux-gnu” and it made no difference.

My test program (sample.cpp), which has a deliberate memory leak to exercise Asan, is

#include <string>

bool f()
{
    auto s = new std::string("ABC");

    return (s != nullptr);
}

int main(int ac, char** av)
{
    return static_cast<int>(f());
}

I’m also on the hunt for an LFS-style build-out and came across thread and had “hope” for a tiny bit…
Coming from Cross Compile Any Llvm Component Using Clang Only (No Gcc Requirement)?, I’m still leading to believe (even after going through this thread and examining provided links) that LLVM doesn’t have parity with GCC’s cross-build chicken<->egg solution.

Using some cheats (eg -DCMAKE_C_COMPILER_WORKS (and others)) to get me to start building, the linker always borks trying to get those files that don’t exist yet at [ 96%] Linking CXX shared library ../linux/libclang_rt.asan-arm.so. Even if you don’t use a “*-*-gnu*” triple.
I’ve heard mention to use rt-compiler, but that’s the chicken<->egg issue right there.
I can’t build it without a sysroot already being built-out, while LFS builds out the sysroot without any need for any pre-existing arch-based binaries.
Having looked at fuchsia, it appears that pre-built sysroots are downloaded and installed first, which in my view defeats the objective of bootstrapping.

My desire is to see LLVM reach parity with GCC, where you can cross-build without the need for any pre-existing sysroot for the target arch. You just build out the sysroot yourself.
That way there will no longer be a need for both compilers, and the only need would be like the libc of choice and the kernel (that would be part of the cross build-out of the sysroot).
Basically it’d be like III.5 of Linux From Scratch where the LLVM bootstrap would take the place of binutils and gcc part1, then we’d use the freshly built llvm to finish with the libc source of our choice (knowing llvm currently can’t do glibc), then I figure the 2nd part of the LLVM boostrap would finish up with compiler-rt and libunwind.

EDIT0: One other notable change is the configure automatic finding of <arch>-<bins> (eg arm-unknown-linux-gnueabi-ar), so we’d have to manually point to llvm-ar via the env-var AR. Not sure how that could be nicely handled, and probably lies with the configure itself.

I’m having same problem with build of native llvm 17 and 18 on RHEL 7, 8.

I went around compiler does not support '-fuse-ld=mold'
by setting PATH to .../libexec/mold/ld (symlink to actual linker mold provides for similar cases).

This fixed compilation for me, but then check-cxx check-cxxabi check-unwind also failed because of inability to find crtbeginS.o and I didn’t sort this (so far).

Hmm. Sounds like a problem for bootstrapping glibc with clang on my phone. Where are the docs for bootstrapping gcc/glibc in these post-CVS days? (I don’t mind if gcc is technically a cross-compiler.)

I suppose I could use the instructions at “build-many-glibcs.py”
Everything I know about glibc | MaskRay assuming I have enough space left…

FWIW, bootstrapping glibc is a bit more involved, but I have fairly straightforward scripts that set up musl cross sysroots from scratch, with compiler-rt/libunwind/libcxx - see the post in the other thread at Cross Compile Any Llvm Component Using Clang Only (No Gcc Requirement)? - #30 by mstorsjo.