Dear community,
Recently I discovered that llvm gold linker plugin (LLVMgold.so) doesn’t add llvm instrinsics symbols to the linker symbol table. I do not claim that something is necessary wrong, just want to share my observations with the community.
Brief summary
If I create a static library with a custom version of ‘exp()’ math function and link it as follows:
$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm
exp gets resolved from libm instead of my static library (libexp.a) even though the static library appears first on the link command line.
Reproducer
- As a reproducer I built a tiny static library (with the name libexp.a) with exp function defined:
$ cat exp.c
double exp (double x)
{
return x;
}
$ clang –c exp.c
$ ar -r libexp.a exp.o
$ nm -S libexp.a | grep exp
0000000000000000 0000000000000010 T exp
- I have another small piece of code which will use ‘exp’ from my library (libexp.a):
$ cat a.c
#include <math.h>
int main()
{
return exp(1.0);
}
$ clang -c -O0 -ffast-math -flto -o a.o a.c
After rC318193 we started lowering math libcalls to equivalent LLVM intrinsics:
$ llvm-dis a.o -o - | grep llvm.exp
%2 = call fast double @llvm.exp.f64(double 1.000000e+00)
- I link my library (libexp.a) statically but link libm dynamically (as a fallback option):
$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic –lm
$ nm -S a.out | grep exp
U exp@@GLIBC_2.2.5
So, exp symbol was taken from libm, not from my static library. Notice, that ‘llvm.exp.f64’ should be lowered into __exp_finite (see https://bugs.llvm.org/show_bug.cgi?id=35672#c9 ).
To understand why this happened I traced ‘exp’ symbol:
$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm -Wl,–trace-symbol=exp
/usr/lib64/libm.so: definition of exp
/tmp/lto-llvm-8f9b9a.o: reference to exp
How linker works in those cases?
In LTO build linker works in 2 phases:
- Runs through all the inputs and identifies which of them should be processed by the plugin. But while doing that linker already starts filling it’s symbol table.
Plugin at that point knows what symbols are defined and what referenced by the input files with bitcode inside (because it knows how to parse it). So, plugin reports that symbols to the linker.
- After all inputs are read linker signals to the plugin to do it’s magic.
We run LTO and produce one “fat” relocatable object file and feed it back to the linker.
Linker finishes it’s job as usual.
More details of this process are described here: whopr/driver - GCC Wiki .
If we apply the process described above to our case, we will understand what happened:
1st pass:
We see a.o. It is claimed by the plugin. But plugin ignored llvm.exp.f64 intrinsic, so ‘exp’ symbol was not reported to the linker as referenced symbol.
We see static linking of libexp. We are linking static libraries only if someone referenced any symbols defined in it, so we skip libexp (no one referenced ‘exp’).
We see dynamic linking of libm. We will link to it unconditionally, so linker captures all symbols in libm. It found definition of ‘exp’ and insert it to the linker symbol table.
We run LTO stage. /tmp/lto-llvm-8f9b9a.o was added to the linker inputs.
2nd pass:
There is a reference to exp in /tmp/lto-llvm-8f9b9a.o, but we have it’s definition already (in libm).
We see libexp again, but it’s too late, we already have exp defined in libm, and it won’t be overridden.
I confirmed all the above by adding debug prints in LLVMGold plugin (LLVMgold.so) and built debug version of binutils (added debug prints to gold linker).
Behavior is similar for gold and bfd linkers.
This problem goes away if I add ‘-u exp’ to the linker command. This option adds undefined symbol ‘exp’ to the linker symbol right at the beginning of the process, so it will be taken in 1st pass from libexp. But of course, it can’t be a solution to this issue.
How it affects us?
We have our own runtime library which provides implementation for math functions like exp, log, etc. So, we use it as a replacement for libm.
We link libm dynamically because not every distribution is supplied with static version of libm.
It’s not so trivial to report symbols that correspond to llvm intrinsics because we don’t know for sure to which exact symbols they will be expanded. For example, in rL322087 we started lowering llvm.exp.* to “__exp_finite” if –ffast-math (or -Ofast) is provided.
Any thoughts on this?
-Denis Bakhvalov
Employee of Intel, compiler development team.