Simple? Linking issue when cross-compiling for AArch64 on X86

Hi all,

I have what seems like a very simple issue for which I’ve found almost no useful information online after quite a lot of reading mailing list threads and etc.

I’ve built LLVM-11 from a git checkout llvmorg-11.0.0 using the following CMAKE options.

cmake -G Ninja ../llvm-project/llvm -DCMAKE_{C_COMPILER=clang,CXX_COMPILER=clang++,BUILD_TYPE="MinSizeRel",INSTALL_PREFIX=$(realpath ../llvm-root/)} -DLLVM_{TARGETS_TO_BUILD="X86;AArch64",INCLUDE_TOOLS=ON,INCLUDE_EXAMPLES=OFF,INCLUDE_TESTS=OFF,INCLUDE_BENCHMARKS=OFF,ENABLE_PROJECTS="clang;flang;compiler-rt;openmp",PARALLEL_COMPILE_JOBS=32,PARALLEL_LINK_JOBS=4,ENABLE_BINDINGS=OFF} -DBUILD_SHARED_LIBS=ON

When I try to compile a simple test program with the resulting clang, I see something different from my system installation (clang 10).

[andrew@baltar ~]$ cat test.c 
int main () {}
[andrew@baltar ~]$ clang --target=aarch64-linux-gnu --sysroot=/usr/aarch64-linux-gnu test.c -o test
[andrew@baltar ~]$ /data/andrew/llvm-root/bin/clang --target=aarch64-linux-gnu --sysroot=/usr/aarch64-linux-gnu test.c -o test
/usr/bin/aarch64-linux-gnu-ld: cannot find crtbegin.o: No such file or directory
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

Those objects are present in /usr/lib/gcc/aarch64-linux-gnu/10.2.0. However, if I just add -L/usr/lib/gcc/aarch64-linux-gnu/10.2.0 or even --gcc-toolchain=/usr/lib/gcc/aarch64-linux-gnu/10.2.0/, the problem persists. How do I actually tell the linker invoked by clang where to look for these objects, and how come they aren’t in the sysroot (/usr/<target>), but rather in /usr/lib/gcc/<target>?

I’ve been stumped by this and I can’t find a solution online, just other folks with similar-ish problems in dead mailing list threads. Would really appreciate some pointers if anyone notices anything wrong! I suspect I need to configure the linker (aarch64-linux-gnu-ld in this case) somehow, but I’m not sure how (and I’m not sure why the -L above doesn’t work because that directory has those objects in it!)

Since the system installation of clang picks up those objects with the same invocation, I suspect I might have missed some configuration option in my build. Is there more to building LLVM for multiple targets than just adding them to the LLVM_TARGETS_TO_BUILD= option?

This option only enables the LLVM backend: i.e. you can emit .o objects for a given target. It does not configure the toolchain though.

A key difference between these two invocations is that clang will looks for relative paths by default to find path it expects to these libraries. You should try these two invocations with -v again to see the difference.
If I remember correctly, using -B is the way to achieve this: it’ll tell clang where to look for these “implicit default path”.

1 Like

Ahhh, I see! Thanks Mehdi, this was exactly what I was missing.

Passing -v, I see that some objects in the linking step are missing the relative path traversal at the beginning:

[andrew@baltar ~]$ /data/andrew/llvm-root/bin/clang -v --target=aarch64-linux-gnu --sysroot=/usr/aarch64-linux-gnu test.c -o test
clang version 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d457092ed932768724a6f06)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /data/andrew/llvm-root/bin
 "/data/andrew/llvm-root/bin/clang-11" -cc1 -triple aarch64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model static -mframe-pointer=non-leaf -fmath-errno -fno-rounding-math -mconstructor-aliases -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /data/andrew/llvm-root/lib/clang/11.0.0 -isysroot /usr/aarch64-linux-gnu -internal-isystem /usr/aarch64-linux-gnu/usr/local/include -internal-isystem /data/andrew/llvm-root/lib/clang/11.0.0/include -internal-externc-isystem /usr/aarch64-linux-gnu/include -internal-externc-isystem /usr/aarch64-linux-gnu/usr/include -fdebug-compilation-dir /home/andrew -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/test-a9609c.o -x c test.c
clang -cc1 version 11.0.0 based upon LLVM 11.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/local/include"
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /data/andrew/llvm-root/lib/clang/11.0.0/include
 /usr/aarch64-linux-gnu/include
End of search list.
 "/usr/bin/aarch64-linux-gnu-ld" --sysroot=/usr/aarch64-linux-gnu -EL --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o test /usr/aarch64-linux-gnu/lib/crt1.o /usr/aarch64-linux-gnu/lib/crti.o crtbegin.o -L/usr/aarch64-linux-gnu/lib/../lib64 -L/usr/aarch64-linux-gnu/lib /tmp/test-a9609c.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed crtend.o /usr/aarch64-linux-gnu/lib/crtn.o
/usr/bin/aarch64-linux-gnu-ld: cannot find crtbegin.o: No such file or directory
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

Adding the missing path prefix with -B fixes this

[andrew@baltar ~]$ /data/andrew/llvm-root/bin/clang -v --target=aarch64-linux-gnu --sysroot=/usr/aarch64-linux-gnu -B/usr/lib/gcc/aarch64-linux-gnu/10.2.0/ test.c -o test
clang version 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d457092ed932768724a6f06)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /data/andrew/llvm-root/bin
 "/data/andrew/llvm-root/bin/clang-11" -cc1 -triple aarch64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model static -mframe-pointer=non-leaf -fmath-errno -fno-rounding-math -mconstructor-aliases -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /data/andrew/llvm-root/lib/clang/11.0.0 -isysroot /usr/aarch64-linux-gnu -internal-isystem /usr/aarch64-linux-gnu/usr/local/include -internal-isystem /data/andrew/llvm-root/lib/clang/11.0.0/include -internal-externc-isystem /usr/aarch64-linux-gnu/include -internal-externc-isystem /usr/aarch64-linux-gnu/usr/include -fdebug-compilation-dir /home/andrew -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/test-3309dc.o -x c test.c
clang -cc1 version 11.0.0 based upon LLVM 11.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/local/include"
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /data/andrew/llvm-root/lib/clang/11.0.0/include
 /usr/aarch64-linux-gnu/include
End of search list.
 "/usr/bin/aarch64-linux-gnu-ld" --sysroot=/usr/aarch64-linux-gnu -EL --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o test /usr/aarch64-linux-gnu/lib/crt1.o /usr/aarch64-linux-gnu/lib/crti.o /usr/lib/gcc/aarch64-linux-gnu/10.2.0/crtbegin.o -L/usr/aarch64-linux-gnu/lib/../lib64 -L/usr/aarch64-linux-gnu/lib /tmp/test-3309dc.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/aarch64-linux-gnu/10.2.0/crtend.o /usr/aarch64-linux-gnu/lib/crtn.o
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
/usr/bin/aarch64-linux-gnu-ld: cannot find -lgcc
clang-11: error: linker command failed with exit code 1 (use -v to see invocation)

I don’t see the -B option anywhere in man clang, and the -B option in man ld looks a bit different to this, so maybe this should be in the bugzilla as a docs issue if it isn’t already (I have a bugzilla account so I’ll search and add it if I can’t find any similar issue pending).

The remaining error seems to be caused by the fact that my system’s GCC toolchain only provides libgcc.a and no libgcc.o, but I suspect that’s an issue with how my aarch64-linux-gnu-gcc package got built. The host GCC is set up the same way.

[andrew@baltar ~]$ pacman -Ql gcc | grep 10.2.0/libgcc
gcc /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc.a
gcc /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0/libgcc_eh.a
[andrew@baltar ~]$ pacman -Ql aarch64-linux-gnu-gcc | grep 10.2.0/libgcc
aarch64-linux-gnu-gcc /usr/lib/gcc/aarch64-linux-gnu/10.2.0/libgcc.a
aarch64-linux-gnu-gcc /usr/lib/gcc/aarch64-linux-gnu/10.2.0/libgcc_eh.a

I don’t suppose clang has a handy --with-static-libgcc option or anything like that? :slight_smile:

Thanks so much for your help! I’ve read what feels like dozens of posts and mailing list replies about issues similar to this and I haven’t seen the -B option mentioned.

Oh, my bad: this second issue is actually fixed by adding a -L pointing to the same directory so that -lgcc is interpreted the right way by the linker:

[andrew@baltar ~]$ /data/andrew/llvm-root/bin/clang -v --target=aarch64-linux-gnu --sysroot=/usr/aarch64-linux-gnu -B/usr/lib/gcc/aarch64-linux-gnu/10.2.0/ -L/usr/lib/gcc/aarch64-linux-gnu/10.2.0/ test.c -o test
clang version 11.0.0 (https://github.com/llvm/llvm-project.git 176249bd6732a8044d457092ed932768724a6f06)
Target: aarch64-unknown-linux-gnu
Thread model: posix
InstalledDir: /data/andrew/llvm-root/bin
 "/data/andrew/llvm-root/bin/clang-11" -cc1 -triple aarch64-unknown-linux-gnu -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -discard-value-names -main-file-name test.c -mrelocation-model static -mframe-pointer=non-leaf -fmath-errno -fno-rounding-math -mconstructor-aliases -target-cpu generic -target-feature +neon -target-abi aapcs -fallow-half-arguments-and-returns -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /data/andrew/llvm-root/lib/clang/11.0.0 -isysroot /usr/aarch64-linux-gnu -internal-isystem /usr/aarch64-linux-gnu/usr/local/include -internal-isystem /data/andrew/llvm-root/lib/clang/11.0.0/include -internal-externc-isystem /usr/aarch64-linux-gnu/include -internal-externc-isystem /usr/aarch64-linux-gnu/usr/include -fdebug-compilation-dir /home/andrew -ferror-limit 19 -fno-signed-char -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/test-26b9f4.o -x c test.c
clang -cc1 version 11.0.0 based upon LLVM 11.0.0 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/local/include"
ignoring nonexistent directory "/usr/aarch64-linux-gnu/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /data/andrew/llvm-root/lib/clang/11.0.0/include
 /usr/aarch64-linux-gnu/include
End of search list.
 "/usr/bin/aarch64-linux-gnu-ld" --sysroot=/usr/aarch64-linux-gnu -EL --eh-frame-hdr -m aarch64linux -dynamic-linker /lib/ld-linux-aarch64.so.1 -o test /usr/aarch64-linux-gnu/lib/crt1.o /usr/aarch64-linux-gnu/lib/crti.o /usr/lib/gcc/aarch64-linux-gnu/10.2.0/crtbegin.o -L/usr/lib/gcc/aarch64-linux-gnu/10.2.0/ -L/usr/aarch64-linux-gnu/lib/../lib64 -L/usr/aarch64-linux-gnu/lib /tmp/test-26b9f4.o -lgcc --as-needed -lgcc_s --no-as-needed -lc -lgcc --as-needed -lgcc_s --no-as-needed /usr/lib/gcc/aarch64-linux-gnu/10.2.0/crtend.o /usr/aarch64-linux-gnu/lib/crtn.o

Problem solved! It feels a little awkward to have to pass -B/path -L/path but everything works now. Thanks again Mehdi for your help!

I don’t know how man clang is setup, I tend to rely on clang --help (and clang --help-hidden).