I have been unable to build clang with cross-compile runtimes for ARM64 due to linker problems. If I let it use gnu ld (2.34), the link fails because the 18GiB RAM of my Ubuntu 20.04 VM is not enough. If I configure to use lld, linking runtimes dies saying “-fuse-ld=lld” is not an option of the host compiler.
Building llvm-project 16.0.5. CMake 3.16.3. GNU compiler tools 9.4.0.
...
[5555/5568] Performing configure step for 'runtimes-X86'
CMake Warning at CMakeLists.txt:4 (message):
Your CMake version is 3.16.3. Starting with LLVM 17.0.0, the minimum
version of CMake required to build LLVM will become 3.20.0, and using an
older CMake will become an error. Please upgrade your CMake to at least
3.20.0 now to avoid issues in the future!
-- The C compiler identification is Clang 16.0.5
-- The CXX compiler identification is Clang 16.0.5
-- The ASM compiler identification is Clang
-- Found assembler: /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/./bin/clang
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - failed
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - failed
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE)
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.10")
-- Could NOT find Terminfo (missing: Terminfo_LIBRARIES Terminfo_LINKABLE)
-- Could NOT find ZLIB (missing: ZLIB_LIBRARY) (found version "1.2.11")
-- Performing Test LLVM_RUNTIMES_LINKING_WORKS
-- Performing Test LLVM_RUNTIMES_LINKING_WORKS - Failed
-- Performing Test CXX_SUPPORTS_UNWINDLIB_EQ_NONE_FLAG
-- Performing Test CXX_SUPPORTS_UNWINDLIB_EQ_NONE_FLAG - Failed
-- Performing Test CXX_SUPPORTS_NOSTDLIBXX_FLAG
-- Performing Test CXX_SUPPORTS_NOSTDLIBXX_FLAG - Failed
-- Performing Test CXX_SUPPORTS_NOSTDINCXX_FLAG
-- Performing Test CXX_SUPPORTS_NOSTDINCXX_FLAG - Failed
-- Linker detection: LLD
-- Performing Test CXX_SUPPORTS_CUSTOM_LINKER
-- Performing Test CXX_SUPPORTS_CUSTOM_LINKER - Failed
CMake Error at /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/llvm/cmake/modules/HandleLLVMOptions.cmake:320 (message):
Host compiler does not support '-fuse-ld=lld'
Call Stack (most recent call first):
CMakeLists.txt:157 (include)
-- Configuring incomplete, errors occurred!
See also "/home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/runtimes/runtimes-X86-bins/CMakeFiles/CMakeOutput.log".
See also "/home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/runtimes/runtimes-X86-bins/CMakeFiles/CMakeError.log".
[5559/5568] Creating library symlink lib/liblldbIntelFeatures.so
FAILED: runtimes/runtimes-X86-stamps/runtimes-X86-configure
CMake Error at /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/llvm/cmake/modules/HandleLLVMOptions.cmake:320 (message):
Host compiler does not support '-fuse-ld=lld'
Call Stack (most recent call first):
CMakeLists.txt:157 (include)
This can fail either because the compiler doesn’t know about the option, or when CMake tried to compile an example with that option, doing so failed. Most commonly because lld is not installed, or it is not available as ld.lld.
If you have apt or equivalent, sudo apt install lld should get you that. Any recent version of it should be fine.
The test that CMake does is something like <compiler> test.c -fuse-ld=lld -o /dev/null. If you can do that then CMake should be ok with it.
Though 18GiB of RAM seems very high for a single link job, but perhaps you have hit some corner case in ld in which case you’re doing the correct thing of trying lld instead.
I appear to be set up right. I can compile using -fuse-ld=lld from the command line using g++ and clang++ 16.0.5 (an earlier installation that does not have the runtimes I want). So, I am still clueless why the complaint. It is specifically when building the runtimes with the newly built compiler that I get this error.
Right I missed that. In that case you built lld so I’m surprised it didn’t just use that. That’s how our prebuilt llvm packages work, the user doesn’t have to apt install lld seperately.
At this point I’d try to find the command line it’s trying, which means debugging the test which apparently prints to a log file: [CMake] Debugging check_cxx_source_compiles
CMake should find the right one, but double check exactly which clang 16.0.5 it’s finding. Ideally remove the old install and reset your PATH and such.
You could also confirm that there is an lld in _build/bin just in case. Sometimes I configure with many components and only build one, then forget and try to use an old copy of one of the others.
LLVM_ENABLE_LLD:BOOL
This option is equivalent to -DLLVM_USE_LINKER=lld, except during a 2-stage build where a dependency is added from the first stage to the second ensuring that lld is built before stage2 begins.
LLVM_RUNTIME_TARGETS option takes target triple as a value so in your case the subbuild is trying to use X86 as a triple and failing.
The -fuse-ld=lld check failing is usually a red herring, in your log output you can see that all other checks prior to it are failing as well, the linker check just happens to be the first one that terminates the CMake execution. I’d check runtimes/runtimes-X86-bins/CMakeFiles/CMakeError.log to get a better understanding of what’s going on, but it’s likely that the compiler cannot find a usable sysroot for the X86 target.
And got this error in the logs, prompting the red herring complaint about lld:
Performing C++ SOURCE FILE Test CXX_SUPPORTS_CUSTOM_LINKER failed with the following output:
Change Dir: /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/runtimes/runtimes-aarch64-unknown-linux-gnu-bins/CMakeFiles/CMakeTmp
Run Build Command(s):/usr/bin/ninja cmTC_4aaa3 && [1/2] Building CXX object CMakeFiles/cmTC_4aaa3.dir/src.cxx.o
[2/2] Linking CXX executable cmTC_4aaa3
FAILED: cmTC_4aaa3
: && /home/yrqfwq4/workflow/one-off/llvm-project-release-16.x/_build/./bin/clang++ --target=aarch64-unknown-linux-gnu -DCXX_SUPPORTS_CUSTOM_LINKER -fuse-ld=lld CMakeFiles/cmTC_4aaa3.dir/src.cxx.o -o cmTC_4aaa3 && :
ld.lld: error: cannot open Scrt1.o: No such file or directory
ld.lld: error: cannot open crti.o: No such file or directory
ld.lld: error: cannot open crtbeginS.o: No such file or directory
ld.lld: error: unable to find library -lstdc++
ld.lld: error: unable to find library -lm
ld.lld: error: unable to find library -lgcc_s
ld.lld: error: unable to find library -lgcc
ld.lld: error: unable to find library -lc
ld.lld: error: unable to find library -lgcc_s
ld.lld: error: unable to find library -lgcc
ld.lld: error: cannot open crtendS.o: No such file or directory
ld.lld: error: cannot open crtn.o: No such file or directory
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
Source file was:
int main() { return 0; }
So what am I still not doing right to build the Aarch64 cross-compile target? I’m not 100% sure I got the target triple right for it.
If you don’t have a standard library available you need to tell CMake to skip those steps via CMAKE_C(XX)_COMPILER_WORKS for the runtimes parts of the build, though I thought we did that automatically in the -DLLVM_ENABLE_RUNTIMES case.
I tried setting -DCMAKE_C_COMPILER_WORKS=1 and -DCMAKE_CXX_COMPILER_WORKS=1, but it didn’t change anything about the failure.
Am I leaving something out in my configuration that would resolve the standard library not being found?
I’ve read everything I could find, but I’m frustrated that there aren’t clear instructions for how to build clang so that it can cross-compile for arm64 from x86_64, complete with runtimes including santizers (I will specifically need Asan). I’m curious how anyone gets it working.
To build a cross-compiler, you want to specify -DLLVM_DEFAULT_TARGET_TRIPLE=your-target-triple or you’ll end up with a target=host non-cross compiler. It will still be able to cross-compile, but it won’t do so by default.
I don’t know whether that will help with your runtimes/sanitizer problems, I don’t ever build those.
Ok, this is specifically because you use -DLLVM_USE_LINKER=lld (or -DLLVM_ENABLE_LLD, which sets LLVM_USE_LINKER=lld). The runtimes build passes this through, but the check for the variable is a check_cxx_source_compiles, which needs a full working C++ sysroot that’s clearly not going to be present in the cross-compiling case. We just need an analogue to force-setting CMAKE_CXX_COMPILER_WORKS. Currently testing exposing an LLVM_LINKER_WORKS.
Fixing that gets further but things are still heavily broken because we’re configuring things before we’ve built other projects and checking properties by linking. In order:
LLVM_USE_LINKER gets passed through (in two places) and doesn’t work when set, so needs LLVM_LINKER_WORKS
libunwind checks for -fno-exceptions outside of set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) and fails again because there’s no libc++, among other things
HandleLLVMOptions warns that -fPIC isn’t supported because it tries to link (both C and C++)
libunwind spews warnings and errors during its build, thinking intptr_t is a plain int not a long and failing to find asm/unistd.h
Clearly the runtimes build expects to find a working set of C headers and libc for that OS (point 4), but 1-3 have nothing to do with that and require you to already have a libc++. 1 is easy to fix. 2 sounds like an ordering problem. 3 needs a bit more thought on whether it’s safe to make that just check compile rather than compile+link. 4 may or may not go away with working OS headers. Who knows what comes next. But it’s clear there’s a lot of brokenness here if you want to use -DLLVM_ENABLE_RUNTIMES for cross-compiling and that you instead need to build the projects one by one in the right order with the right flags yourself, which somewhat defeats the point.
The difference is that we pass additional flags and use different defaults, for example we pass explicit sysroot, we use LLD as a linker, compiler-rt builtins as compiler runtimes, libunwind as the unwinder and configure each runtime build as such by setting the RUNTIMES_${target}_${option} variables.
I also build various other configurations locally during development, but the full matrix of possible configurations is too large for me to test manually so it’s entirely possible that there are some configurations that don’t work and if that’s the case, we should fix them.
Unless you use an explicit sysroot, the behavior may also depend on your host installation. For example if you don’t have the aarch64-unknown-linux-gnu libraries installed on your host and you set -DLLVM_RUNTIME_TARGETS=aarch64-unknown-linux-gnu, I’d expect things are to break which may actually be the issue here (that’s why LLD is failing to find libraries such as libgcc, libc or libstdc++).
In order to better assist, it’d be helpful to explain what you’re trying to achieve, and I can try and suggest the CMake flags you might need.
I made a lot of headway by installing the following in my Ubuntu 20.04 VM:
g+±aarch64-linux-gnu
gcc-aarch64-linux-gnu
binutils-aarch64-linux-gnu
gcc-multilib
g+±multilib
This made thinks work well enough that my LLVM/clang build completes without error.
I ran “apt install clang” as well, to shake things down with the dependencies using what should be a known-good clang.
I can cross-compile to aarch64 successfully with aarch64-linux-gnu-g++, but cross-compiling with “clang --target=aarch64-linux-gnu XXX.cpp” gets a missing header error for bits/c++-config.h when including headers such as <string>.
In file included from sample.cpp:1:
/usr/bin/../lib/gcc-cross/aarch64-linux-gnu/9/../../../../include/c++/9/string:38:10: fatal error: 'bits/c++config.h' file not found
#include <bits/c++config.h>
^~~~~~~~~~~~~~~~~~
1 error generated.
I tried adding “–sysroot=/usr/aarch64-linux-gnu” and it made no difference.