Cross compiling fails with linker errors when using pthreads

Clang Frontend Using Clang
1

Hi,

I’m trying to cross compile to an older linux distribution, to get wider glibc compatibility for my binary.

I’ve installed a debian 10 sysroot in /tmp/deb10. The host is a current Arch linux. In a chroot environment I’ve built a recent clang and lld. Within the chroot I’m able to succesfully compile and link.

My goal is to compile from outside the sysroot, using --sysroot. This works for the simple case, but fails as soon as I link pthreads.

Test file:

[joostn@arch]$ cat /tmp/deb10/home/joostn/checkpthread.c
#include <pthread.h>
#include <stdio.h>

static void* test_func(void* data)
{
  return data;
}

int main(void)
{
  puts("Hi\n");

  pthread_t thread;
  pthread_create(&thread, NULL, test_func, NULL);
  pthread_detach(thread);
  pthread_cancel(thread);
  pthread_join(thread, NULL);
  pthread_atfork(NULL, NULL, NULL);
  pthread_exit(NULL);

  return 0;
}

Inside the chroot I can compile a working binary:

$ clang --sysroot=/ /home/joostn/checkpthread.c -pthread -fuse-ld=lld -v -o /home/joostn/checkpthread
clang version 17.0.4
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/bin
Found candidate GCC installation: /lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /lib/gcc/x86_64-linux-gnu/8
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/local/bin/clang-17" -cc1 -triple x86_64-unknown-linux-gnu -emit-obj -mrelax-all -dumpdir /home/joostn/checkpthread- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name checkpthread.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -v -fcoverage-compilation-dir=/home/joostn -resource-dir /usr/local/lib/clang/17 -isysroot / -internal-isystem /usr/local/lib/clang/17/include -internal-isystem /usr/local/include -internal-isystem /lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdebug-compilation-dir=/home/joostn -ferror-limit 19 -pthread -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/checkpthread-fd9db6.o -x c /home/joostn/checkpthread.c
clang -cc1 version 17.0.4 based upon LLVM 17.0.4 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/lib/clang/17/include
 /usr/local/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/local/bin/ld.lld" --sysroot=/ -pie --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /home/joostn/checkpthread /lib/x86_64-linux-gnu/Scrt1.o /lib/x86_64-linux-gnu/crti.o /lib/gcc/x86_64-linux-gnu/8/crtbeginS.o -L/lib/gcc/x86_64-linux-gnu/8 -L/lib/gcc/x86_64-linux-gnu/8/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/lib -L/usr/lib /tmp/checkpthread-fd9db6.o -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /lib/gcc/x86_64-linux-gnu/8/crtendS.o /lib/x86_64-linux-gnu/crtn.o

Trying to do the same from the host, using --sysroot, it’s giving linker errors:

[joostn@tootsarch]$ clang --sysroot=/tmp/deb10 /tmp/deb10/home/joostn/checkpthread.c -pthread -ldl -fuse-ld=lld -v -o /tmp/deb10/home/joostn/checkpthread
clang version 16.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /tmp/deb10/lib/gcc/x86_64-linux-gnu/8
Selected GCC installation: /tmp/deb10/lib/gcc/x86_64-linux-gnu/8
Candidate multilib: .;@m64
Selected multilib: .;@m64
 "/usr/bin/clang-16" -cc1 -triple x86_64-pc-linux-gnu -emit-obj -mrelax-all -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name checkpthread.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=gdb -v -fcoverage-compilation-dir=/home/joostn/devel/wxwprojects_ptgui13 -resource-dir /usr/lib/clang/16 -isysroot /tmp/deb10 -internal-isystem /usr/lib/clang/16/include -internal-isystem /tmp/deb10/usr/local/include -internal-isystem /tmp/deb10/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/include -internal-externc-isystem /tmp/deb10/usr/include/x86_64-linux-gnu -internal-externc-isystem /tmp/deb10/include -internal-externc-isystem /tmp/deb10/usr/include -fdebug-compilation-dir=/home/joostn/devel/wxwprojects_ptgui13 -ferror-limit 19 -pthread -stack-protector 2 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/checkpthread-0f1163.o -x c /tmp/deb10/home/joostn/checkpthread.c
clang -cc1 version 16.0.6 based upon LLVM 16.0.6 default target x86_64-pc-linux-gnu
ignoring nonexistent directory "/tmp/deb10/lib/gcc/x86_64-linux-gnu/8/../../../../x86_64-linux-gnu/include"
ignoring nonexistent directory "/tmp/deb10/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/clang/16/include
 /tmp/deb10/usr/local/include
 /tmp/deb10/usr/include/x86_64-linux-gnu
 /tmp/deb10/usr/include
End of search list.
 "/usr/bin/ld.lld" --sysroot=/tmp/deb10 -pie --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o /tmp/deb10/home/joostn/checkpthread /tmp/deb10/lib/x86_64-linux-gnu/Scrt1.o /tmp/deb10/lib/x86_64-linux-gnu/crti.o /tmp/deb10/lib/gcc/x86_64-linux-gnu/8/crtbeginS.o -L/tmp/deb10/lib/gcc/x86_64-linux-gnu/8 -L/tmp/deb10/lib/gcc/x86_64-linux-gnu/8/../../../../lib64 -L/tmp/deb10/lib/x86_64-linux-gnu -L/tmp/deb10/lib/../lib64 -L/tmp/deb10/usr/lib/x86_64-linux-gnu -L/tmp/deb10/usr/lib/../lib64 -L/tmp/deb10/lib -L/tmp/deb10/usr/lib /tmp/checkpthread-0f1163.o -ldl -lgcc --as-needed -lgcc_s --no-as-needed -lpthread -lc -lgcc --as-needed -lgcc_s --no-as-needed /tmp/deb10/lib/gcc/x86_64-linux-gnu/8/crtendS.o /tmp/deb10/lib/x86_64-linux-gnu/crtn.o
ld.lld: error: undefined symbol: _dl_cpuclock_offset
>>> referenced by nptl-init.c:279
>>>               nptl-init.o:(__pthread_initialize_minimal_internal) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a

ld.lld: error: undefined symbol: _dl_pagesize
>>> referenced by nptl-init.c:396
>>>               nptl-init.o:(__pthread_initialize_minimal_internal) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a
>>> referenced by nptl-init.c:405
>>>               nptl-init.o:(__pthread_initialize_minimal_internal) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a
>>> referenced by nptl-init.c:442
>>>               nptl-init.o:(__pthread_get_minstack) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a

ld.lld: error: undefined symbol: _dl_init_static_tls
>>> referenced by nptl-init.c:421
>>>               nptl-init.o:(__pthread_initialize_minimal_internal) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a

ld.lld: error: undefined symbol: _dl_wait_lookup_done
>>> referenced by nptl-init.c:423
>>>               nptl-init.o:(__pthread_initialize_minimal_internal) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a

ld.lld: error: undefined symbol: _dl_stack_flags
>>> referenced by allocatestack.c:526
>>>               pthread_create.o:(__pthread_create_2_1) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a
>>> referenced by allocatestack.c:646
>>>               pthread_create.o:(__pthread_create_2_1) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a

ld.lld: error: undefined symbol: _dl_x86_cpu_features
>>> referenced by elision-conf.c:67 (../sysdeps/unix/sysv/linux/x86/elision-conf.c:67)
>>>               elision-lock.o:(_dl_tunable_set_elision_enable) in archive /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

I’ve also tried using /tmp/deb10/usr/local/bin/clang instead of clang, but this gives exactly the same problem.

If I comment out the pthread calls in the .c file, I get a working binary.

Any help appreciated!

2

Found the culprit!

[joostn@tootsarch]$ ll /tmp/deb10/lib/x86_64-linux-gnu/libpthread*
-rwxr-xr-x 1 root root  146968 15 mrt  2022 /tmp/deb10/lib/x86_64-linux-gnu/libpthread-2.28.so
-rw-r--r-- 1 root root 7112246 15 mrt  2022 /tmp/deb10/lib/x86_64-linux-gnu/libpthread.a
lrwxrwxrwx 1 root root      37 15 mrt  2022 /tmp/deb10/lib/x86_64-linux-gnu/libpthread.so -> /lib/x86_64-linux-gnu/libpthread.so.0
lrwxrwxrwx 1 root root      18 15 mrt  2022 /tmp/deb10/lib/x86_64-linux-gnu/libpthread.so.0 -> libpthread-2.28.so
[joostn@tootsarch wxwprojects_ptgui13]$ ll /tmp/deb10/lib/x86_64-linux-gnu/libpthread.so

In the sysroot, the lippthread.so symlink was pointing to an absolute path, so it only worked inside a chroot. From the host the symlink did not resolve correctly, and as a result the linker chose to link against libpthread.a instead of lippthread.so.