Hi, I’m cross posting from here to reach more people possible. I’m trying to compile a C/C++ project that targets an old Linux system. To have the old libraries at hand, I made a copy of the target root fs, and since GCC has hardcoded the library path I cannot really cross-compile against an old version of libc. Therefore I am using Clang + lld. The point is that when linking it complains about
[ 71%] Linking CXX executable single_lidar_listener
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Selected GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Candidate multilib: .;@m32
Selected multilib: .;@m32
"/usr/bin/ld.lld" --sysroot=/home/abertulli/box-root-inst -pie -EL -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o single_lidar_listener /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0 -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../lib -L/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/lib/../lib -L/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/usr/lib/../lib -L/home/abertulli/box-root-inst/lib -L/home/abertulli/box-root-inst/usr/lib CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o libtoojpeg.a -ldl -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
ld.lld: error: undefined symbol: __dlopen
>>> referenced by dlopen.o:(dlopen) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
ld.lld: error: undefined symbol: __dlclose
>>> referenced by dlclose.o:(.text+0x0) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
ld.lld: error: undefined symbol: __dlsym
>>> referenced by dlsym.o:(dlsym) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/single_lidar_listener.dir/build.make:114: single_lidar_listener] Error 1
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/single_lidar_listener.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
Analyzing the content of libc.a as suggested in this answer, I see that these symbols are present. So why can’t Clang find them? Note that sysroot is set, and the GCC installation found seems to be the correct (target-rooted) one. thanks!
my toolchain file
# the name of the target operating system
set(CMAKE_SYSTEM_NAME Linux)
include(CMakePrintHelpers)
cmake_print_variables(CMAKE_DL_LIBS)
# where is the target environment located
set(root_fs_dir /home/abertulli/box-root-inst)
set(CMAKE_FIND_ROOT_PATH ${root_fs_dir})
set(CMAKE_SYSROOT ${root_fs_dir})
# which compilers to use for C and C++
set(CMAKE_C_COMPILER "clang")
set(CMAKE_CXX_COMPILER "clang++")
set(CMAKE_LINKER "ld.lld")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -v")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} --gcc-toolchain=${CMAKE_SYSROOT}/usr")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} --gcc-toolchain=${CMAKE_SYSROOT}/usr")
set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY) # to avoid trying to link and execute test programs
set(CMAKE_C_COMPILER_TARGET arm-unknown-linux-gnueabihf)
set(CMAKE_CXX_COMPILER_TARGET arm-unknown-linux-gnueabihf)
# adjust the default behavior of the FIND_XXX() commands:
# search programs in the host environment
set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# search headers and libraries in the target environment
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
Try to add -ldl to your linker line. Ideally the CMake scripts would detect that the platform needs to link to libdl, but many don’t. The proper way to do this is to add CMAKE_DL_LIBS to link flags.
I think I linked it correctly in the CMakeLists with
target_link_libraries(${PROJECT_NAME} ${CMAKE_DL_LIBS})
but also you can see in the output I posted (my newlines):
"/usr/bin/ld.lld" --sysroot=/home/abertulli/box-root-inst -pie -EL -z
relro -X --hash-style=gnu --build-id --eh-frame-hdr -m
armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o
single_lidar_listener
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o
-L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
-L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../lib
-L/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf
-L/home/abertulli/box-root-inst/lib/../lib
-L/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf
-L/home/abertulli/box-root-inst/usr/lib/../lib
-L/home/abertulli/box-root-inst/lib
-L/home/abertulli/box-root-inst/usr/lib
CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o
CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o
libtoojpeg.a -ldl -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
Try -Wl,-t,-y,__dlopen. libdl.a should define __dlopen.
See Analysis and introspection options in linkers | MaskRay for -t and -y.
You might relink with -Wl,--reproduce=/tmp/rep.tar and upload the tarball in a file host website.
Thanks for the suggestion. Apparently libdl.a gets checked, and a reference is found, but not the definition:
[ 42%] Linking CXX executable single_lidar_listener
Ubuntu clang version 14.0.0-1ubuntu1.1
Target: arm-unknown-linux-gnueabihf
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6
Found candidate GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Selected GCC installation: /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0
Candidate multilib: .;@m32
Selected multilib: .;@m32
"/usr/bin/ld.lld" --sysroot=/home/abertulli/box-root-inst -pie -EL -z relro -X --hash-style=gnu --build-id --eh-frame-hdr -m armelf_linux_eabi -dynamic-linker /lib/ld-linux-armhf.so.3 -o single_lidar_listener /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0 -L/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/../../../../lib -L/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/lib/../lib -L/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf -L/home/abertulli/box-root-inst/usr/lib/../lib -L/home/abertulli/box-root-inst/lib -L/home/abertulli/box-root-inst/usr/lib -t -y __dlopen -t -y __dlopen CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o libtoojpeg.a -ldl -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crti.o
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtbeginS.o
CMakeFiles/single_lidar_listener.dir/single_lidar.cpp.o
CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o)
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o): reference to __dlopen
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o)
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o)
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/libstdc++.so
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libgcc_s.so.1
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libc.so.6
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc_nonshared.a(elf-init.oS)
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/ld-linux-armhf.so.3
/home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libgcc_s.so.1
/home/abertulli/box-root-inst/usr/lib/gcc/arm-linux-gnueabihf/6.3.0/crtendS.o
/home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/crtn.o
ld.lld: error: undefined symbol: __dlopen
>>> referenced by dlopen.o:(dlopen) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
ld.lld: error: undefined symbol: __dlclose
>>> referenced by dlclose.o:(.text+0x0) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
ld.lld: error: undefined symbol: __dlsym
>>> referenced by dlsym.o:(dlsym) in archive /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [CMakeFiles/single_lidar_listener.dir/build.make:114: single_lidar_listener] Error 1
make[1]: *** [CMakeFiles/Makefile2:113: CMakeFiles/single_lidar_listener.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
checking again with
readelf /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a -s
I find
File: /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o)
Symbol table '.symtab' contains 12 entries:
Num: Value Size Type Bind Vis Ndx Name
0: 00000000 0 NOTYPE LOCAL DEFAULT UND
1: 00000000 0 NOTYPE LOCAL DEFAULT 1 $t
2: 00000000 0 NOTYPE LOCAL DEFAULT 5 $d
3: 00000000 130 OBJECT LOCAL DEFAULT 5 __evoke_link_war[...]
4: 00000000 0 SECTION LOCAL DEFAULT 1 .text
5: 00000000 0 SECTION LOCAL DEFAULT 3 .data
6: 00000000 0 SECTION LOCAL DEFAULT 4 .bss
7: 00000000 0 SECTION LOCAL DEFAULT 5 .gnu.warning.dlopen
8: 00000000 0 SECTION LOCAL DEFAULT 6 .note.GNU-stack
9: 00000000 0 SECTION LOCAL DEFAULT 7 .ARM.attributes
10: 00000001 12 FUNC GLOBAL DEFAULT 1 dlopen
11: 00000000 0 NOTYPE GLOBAL DEFAULT UND __dlopen
File: /home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o)
[...]
Maybe the problem is that __dlopen is not defined as a function?
The libraries tar is here
Yes.
--why-extract=a.txt shows that libdl.a(dlopen.o) is extracted to satisfy a reference to dlopen. But __dlopen needed by it is not satisfied by any input file, therefore the error.
reference extracted symbol
home/abertulli/two_lidars/build/CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o) dlopen
home/abertulli/two_lidars/build/CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o) dlclose
home/abertulli/two_lidars/build/CMakeFiles/single_lidar_listener.dir/sick_scan_xd_api_wrapper.c.o home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o) dlsym
home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/Scrt1.o home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc_nonshared.a(elf-init.oS) __libc_csu_init
In newer glibc, some libdl.a symbols have been moved to libc.a. __dlopen is available as a definition.
% nm --quiet -A /usr/lib/x86_64-linux-gnu/libc.a | grep __dlopen
/usr/lib/x86_64-linux-gnu/libc.a:vtables.o: w __dlopen
/usr/lib/x86_64-linux-gnu/libc.a:dlopen.o:00000000000000e0 T ___dlopen
/usr/lib/x86_64-linux-gnu/libc.a:dlopen.o:0000000000000080 T __dlopen
/usr/lib/x86_64-linux-gnu/libc.a:rtld_static_init.o: U __dlopen
Thank you, but I’m not so expert in the linking process, and I can’t understand this:
Shouldn’t Clang already be compiling against libc.a, and anyway, can I do something to make it find the functions resolve the error?
I think this is unlikely an issue of Clang or lld. Your libc doesn’t provide __dlopen.
% tar xf rep.tar
% cd rep
% cat home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc.so # change the absolute paths
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/libc.so.6 home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libc_nonshared.a AS_NEEDED ( home/abertulli/box-root-inst/lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 ) )
% sed -i '/--sysroot/d' response.txt
% arm-linux-gnueabi-ld @response.txt
...
arm-linux-gnueabi-ld: home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlopen.o): in function `dlopen':
(.text+0x8): undefined reference to `__dlopen'
arm-linux-gnueabi-ld: home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlclose.o): in function `dlclose':
(.text+0x0): undefined reference to `__dlclose'
arm-linux-gnueabi-ld: home/abertulli/box-root-inst/usr/lib/arm-linux-gnueabihf/libdl.a(dlsym.o): in function `dlsym':
(.text+0x8): undefined reference to `__dlsym'
GNU ld reports similar errors.
To have the old libraries at hand, I made a copy of the target root fs, and since GCC has hardcoded the library path I cannot really cross-compile against an old version of libc. Therefore I am using Clang + lld. The point is that when linking it complains about
Cross compilation with GCC is indeed more tricky… And its --sysroot= is not that useful in contrast to Clang. If you specify a custom specs file with gcc -specs= you can customize the linker command line. GCC specs: an introduction
gcc -dumpspecs
I see, thank you! So basically GCC uses the specs to find the correct symbols, and that configuration is not used by Clang, so it cannot find them?
But then my question is: why does it work on the native machine? Clang on the target system, when doing native compilation, seems to find the symbols just fine, and those are defined only in libc.a, which should then be searched for correctly? Is there something, when doing cross-compilation / using sysroot, that prevents Clang from searching them?
$ for lib in $(find /opt/box-root-fs -name \*.a) ; do echo $lib ; nm $lib 2> /dev/null | grep __dlopen | grep -v " U " ; done
[...]
/opt/box-root-fs/usr/lib/arm-linux-gnueabihf/libc.a
w __dlopen
00000081 T __dlopen
In other words, Clang should use the libc provided in the sysroot, that does provide __dlopen?
Some guessing:
- Clang doesn’t detect the correct libraries when cross-compiling because the driver isn’t smart enough. By this I mean, there’s probably a pattern that recognizes native libraries that do not extend to independent sysroot environments. If this is true, then this is a driver bug and should be fixed, but we need to have a reproducer that this is the case.
- GCC native versus cross builds are different. This would be a GCC issue, not necessarily a bug. It’s possible (though I’m just guessing) that their driver has similar issues as Clang, in that the code that discovers where libraries, headers and tools are is a mess. In this case, it’s possible (still guessing) that the build system has co-evolved with the driver and created a disparity that GCC “doesn’t see” because it’s hand-crafted.
The main difference above is the age-old Clang-native-cross-compiler versus GCC-combinatorial-build-cross-compiler, where the decisions are taken at run time for Clang and compile time for GCC.
Even if neither of these guesses are true, I hope this can at least enlighten the search for the actual problem.