I’m trying to build clang on x86_64 Ubuntu 20.04 with Asan libraries for AArch64. Having no end of trouble. I feel like I’m peeling an onion. Anyone have any example CMake command lines for this?
Tell me if I understood your goal.
Your host platform is x86_64 Ubuntu 20.04.
You wish to compile a clang and a set of Asan libraries, and both of those will run on AArch64.
Assuming that’s correct, what commands have you tried so far and what results did you get?
I often cross compile an AArch64 lldb from an x86_64 host, so I can adapt that command for you if I know the specifics.
If I understand you, my goal is a little different.
Goal is to build clang and runtime on x86_64 Ubuntu 20.04 to run there and be able to compile AArch64 programs with Asan. I’ll post the cmake command I’m currently using, which fails during building the runtime.
BTW building cmake 16.0.5.
cmake \
-G "CodeBlocks - Ninja" \
-DCMAKE_BUILD_TYPE=${var1} \
${var2}/source/llvm \
-DCMAKE_INSTALL_PREFIX:PATH=${var3} \
-DLLVM_PARALLEL_LINK_JOBS=1 \
-DLLVM_ENABLE_PROJECTS='clang;clang-tools-extra;lldb;lld;polly' \
-DLLVM_ENABLE_RUNTIMES=compiler-rt \
-DLLVM_RUNTIME_TARGETS='X86;AArch64' \
"$@"
What’s the error message?
FWIW It should specify exact triples, e.g. aarch64-linux-android.
I got this working locally, so there may be some missing packages on your machine as I’ve got lots of random stuff installed for cross compiling.
CMake:
cmake \
-G "Ninja" \
-DCMAKE_BUILD_TYPE=Release \
../llvm-project/llvm/ \
-DLLVM_ENABLE_PROJECTS='clang' \
-DLLVM_ENABLE_RUNTIMES=compiler-rt \
-DLLVM_RUNTIME_TARGETS='x86_64-pc-linux-gnu;aarch64-unknown-linux-gnu' \
-DLLVM_TARGETS_TO_BUILD="X86;AArch64"
(TARGETS_TO_BUILD saves some build time as clang won’t be targeting anything else)
I’ve specified the full triples for runtimes. You can get these by running an existing clang:
$ clang -v
clang version 10.0.1-++20200519101712+f79cd71e145-1~exp1~20200519202316.166
Target: x86_64-pc-linux-gnu
<...>
This cmake config will build a new clang, then use that clang to build the runtimes.
I had to install some extra packages to get the required includes for the build:
$ sudo apt install g++-9-aarch64-linux-gnu gcc-9-aarch64-linux-gnu
Then compiled an example:
$ ./bin/clang -target aarch64-unknown-linux-gnu /tmp/test.cpp -o /tmp/test.o -fsanitize=address
And tried a simple buffer overflow example on native AArch64:
$ ./test.o
=================================================================
==287026==ERROR: AddressSanitizer: stack-buffer-overflow on address 0xffff88e00022 at pc 0xaaaae8e57e38 bp 0xffffedf2a570 sp 0xffffedf2a568
WRITE of size 1 at 0xffff88e00022 thread T0
You can also run it from x86 using qemu if you provide the dynamic loader path. Though ASAN seemed to fail with an internal error, but that could be down to many other factors. I’d get native execution working smoothly first.
$ qemu-aarch64-static -L /usr/aarch64-linux-gnu/ /tmp/test.o
AddressSanitizer: CHECK failed: asan_poisoning.cpp:38 "((AddrIsInMem(addr))) != (0)" (0x0, 0x0) (tid=2754746)
<empty stack>
So that’s a configuration that can work, let’s see what you get on your machine.
You can remove the x86 triple here, I misread your post and thought you were looking for both. Though it’s not much extra build time and you’ll be able to compare results if needed.
I tried with full triples and get errors like this:
CMake Error at CMakeLists.txt:249 (add_dependencies):
The dependency target "scudo_standalone" of target "runtimes-test-depends"
does not exist.
scudo_standalone is showing up a lot.
This seems to be where I’m failing. Things are complicated by using some proprietary wrappers for our build system. I have to mask some things that might reveal my client.
[6592/6606] Linking CXX shared library lib/liblldb.so.16.0.5
FAILED: runtimes/runtimes-aarch64-unknown-linux-gnu-stamps/runtimes-aarch64-unknown-linux-gnu-configure
cd XXX/runtimes/runtimes-aarch64-unknown-linux-gnu-bins && /opt/jdx/tools/bin/cmake -DCMAKE_C_COMPILER=XXX/./bin/clang -DCMAKE_CXX_COMPILER=XXX/./bin/clang++ -DCMAKE_ASM_COMPILER=XXX/./bin/clang -DCMAKE_LINKER=XXX/./bin/ld.lld -DCMAKE_AR=XXX/./bin/llvm-ar -DCMAKE_RANLIB=XXX/./bin/llvm-ranlib -DCMAKE_NM=XXX/./bin/llvm-nm -DCMAKE_OBJDUMP=XXX/./bin/llvm-objdump -DCMAKE_OBJCOPY=XXX/./bin/llvm-objcopy -DCMAKE_STRIP=XXX/./bin/llvm-strip -DCMAKE_READELF=XXX/./bin/llvm-readelf -DCMAKE_C_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_CXX_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_ASM_COMPILER_TARGET=aarch64-unknown-linux-gnu -DCMAKE_INSTALL_PREFIX=/usr/clang16 -DLLVM_BINARY_DIR=XXX -DLLVM_CONFIG_PATH=XXX/bin/llvm-config -DLLVM_ENABLE_WERROR=OFF -DLLVM_HOST_TRIPLE=x86_64-unknown-linux-gnu -DLLVM_HAVE_LINK_VERSION_SCRIPT=1 -DLLVM_USE_RELATIVE_PATHS_IN_DEBUG_INFO=OFF -DLLVM_USE_RELATIVE_PATHS_IN_FILES=OFF -DLLVM_LIT_ARGS=-sv -DLLVM_SOURCE_PREFIX= -DPACKAGE_VERSION=16.0.5 -DCMAKE_BUILD_TYPE=Release -DCMAKE_MAKE_PROGRAM=XXX/usr/bin/ninja -DCMAKE_C_COMPILER_LAUNCHER= -DCMAKE_CXX_COMPILER_LAUNCHER= -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCOMPILER_RT_BUILD_BUILTINS=Off -DLLVM_INCLUDE_TESTS=ON -DLLVM_DEFAULT_TARGET_TRIPLE=aarch64-unknown-linux-gnu -DLLVM_ENABLE_PROJECTS_USED=ON -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON -DCMAKE_C_COMPILER_WORKS=ON -DCMAKE_CXX_COMPILER_WORKS=ON -DCMAKE_ASM_COMPILER_WORKS=ON -DCOMPILER_RT_DEFAULT_TARGET_ONLY=ON -DLLVM_RUNTIMES_TARGET=aarch64-unknown-linux-gnu -DHAVE_LLVM_LIT=ON -DLLVM_ENABLE_RUNTIMES=compiler-rt -DLLVM_USE_LINKER= "-GCodeBlocks - Ninja" XXX/source/llvm/runtimes/../../runtimes && XXX/bin/cmake -E touch XXX/runtimes/runtimes-aarch64-unknown-linux-gnu-stamps//runtimes-aarch64-unknown-linux-gnu-configure
[6594/6606] Linking CXX shared library lib/liblldb.so.16.0.5
ninja: build stopped: subcommand failed.
That tell us what failed but not why, which often happens with parallel ninja because the output isn’t in order. You could try again with ninja -j1 so that it’s sequential and see if the error is printed after the failed command.
Otherwise, your best bet from here is to try to replicate the configuration I showed, using publicly available tools. Only once it’s working, use the specific tools you have locally.
Existing uses in llvm all use the full triple so I’m sure that’s the way to use it. You getting that error might actually be a good sign that it’s getting further into the build.
If that is still an issue, you might (not tested it myself) be able to set COMPILER_RT_SANITIZERS_TO_BUILD to not include scudo (https://github.com/llvm/llvm-project/blob/main/compiler-rt/cmake/config-ix.cmake#L706).
I built 16.0.5 without using our local tools. Used your cmake line (with path tweaked). Still failed with unclear reason. Had errors about scudo. I make the change you suggested, but still get errors about scudo. Will post cmake command and build errors.
cmake
-G “Ninja”
-DCMAKE_BUILD_TYPE=Release
…/llvm/
-DLLVM_ENABLE_PROJECTS=‘clang’
-DLLVM_ENABLE_RUNTIMES=‘compiler-rt’
-DCOMPILER_RT_SANITIZERS_TO_BUILD=‘asan;ubsan’
-DLLVM_RUNTIME_TARGETS=‘x86_64-pc-linux-gnu;aarch64-unknown-linux-gnu’
-DLLVM_TARGETS_TO_BUILD=‘X86;AArch64’
– Generated Sanitizer SUPPORTED_TOOLS list on “Linux” is “asan;lsan;msan;tsan;ubsan”
– sanitizer_common tests on “Linux” will run against “asan;lsan;msan;tsan;ubsan”
– Supported architectures for crt: aarch64
– Configuring done
CMake Error at /home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/llvm/cmake/modules/AddLLVM.cmake:1915 (add_dependencies):
The dependency target “scudo_standalone” of target “check-runtimes” does
not exist.
Call Stack (most recent call first):
/home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/llvm/cmake/modules/AddLLVM.cmake:1955 (add_lit_target)
CMakeLists.txt:253 (umbrella_lit_testsuite_end)
CMake Error at CMakeLists.txt:249 (add_dependencies):
The dependency target “scudo_standalone” of target “runtimes-test-depends”
does not exist.
CMake Error at /home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/llvm/cmake/modules/AddLLVM.cmake:1915 (add_dependencies):
The dependency target “scudo_standalone” of target “check-compiler-rt” does
not exist.
Call Stack (most recent call first):
/home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/llvm/cmake/modules/AddLLVM.cmake:1955 (add_lit_target)
/home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/compiler-rt/test/CMakeLists.txt:117 (umbrella_lit_testsuite_end)
CMake Error at /home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/llvm/cmake/modules/AddLLVM.cmake:1915 (add_dependencies):
The dependency target “scudo_standalone” of target “check-gwp_asan” does
not exist.
Call Stack (most recent call first):
/home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/llvm/cmake/modules/AddLLVM.cmake:1981 (add_lit_target)
/home/yrqfwq4/workspace/one-off/llvm-project-release-16.x/compiler-rt/test/gwp_asan/CMakeLists.txt:40 (add_lit_testsuite)
– Generating done
CMake Warning:
Manually-specified variables were not used by the project:
LLVM_CONFIG_PATH
CMake Generate step failed. Build files cannot be regenerated correctly.