LLVM gold plugin do not add llvm instrinsics symbols to the linker symbol table

Dear community,

Recently I discovered that llvm gold linker plugin (LLVMgold.so) doesn’t add llvm instrinsics symbols to the linker symbol table. I do not claim that something is necessary wrong, just want to share my observations with the community.

Brief summary

If I create a static library with a custom version of ‘exp()’ math function and link it as follows:

$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm

exp gets resolved from libm instead of my static library (libexp.a) even though the static library appears first on the link command line.

Reproducer

  1. As a reproducer I built a tiny static library (with the name libexp.a) with exp function defined:

$ cat exp.c

double exp (double x)

{

return x;

}

$ clang –c exp.c

$ ar -r libexp.a exp.o

$ nm -S libexp.a | grep exp

0000000000000000 0000000000000010 T exp

  1. I have another small piece of code which will use ‘exp’ from my library (libexp.a):

$ cat a.c

#include <math.h>

int main()

{

return exp(1.0);

}

$ clang -c -O0 -ffast-math -flto -o a.o a.c

After rC318193 we started lowering math libcalls to equivalent LLVM intrinsics:

$ llvm-dis a.o -o - | grep llvm.exp

%2 = call fast double @llvm.exp.f64(double 1.000000e+00)

  1. I link my library (libexp.a) statically but link libm dynamically (as a fallback option):

$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic –lm

$ nm -S a.out | grep exp

U exp@@GLIBC_2.2.5

So, exp symbol was taken from libm, not from my static library. Notice, that ‘llvm.exp.f64’ should be lowered into __exp_finite (see https://bugs.llvm.org/show_bug.cgi?id=35672#c9 ).

To understand why this happened I traced ‘exp’ symbol:

$ clang -O0 -ffast-math -flto a.o -L. -Wl,-Bstatic -lexp -Wl,-Bdynamic -lm -Wl,–trace-symbol=exp

/usr/lib64/libm.so: definition of exp

/tmp/lto-llvm-8f9b9a.o: reference to exp

How linker works in those cases?

In LTO build linker works in 2 phases:

  1. Runs through all the inputs and identifies which of them should be processed by the plugin. But while doing that linker already starts filling it’s symbol table.

Plugin at that point knows what symbols are defined and what referenced by the input files with bitcode inside (because it knows how to parse it). So, plugin reports that symbols to the linker.

  1. After all inputs are read linker signals to the plugin to do it’s magic.

We run LTO and produce one “fat” relocatable object file and feed it back to the linker.

Linker finishes it’s job as usual.

More details of this process are described here: https://gcc.gnu.org/wiki/whopr/driver .

If we apply the process described above to our case, we will understand what happened:

1st pass:

We see a.o. It is claimed by the plugin. But plugin ignored llvm.exp.f64 intrinsic, so ‘exp’ symbol was not reported to the linker as referenced symbol.

We see static linking of libexp. We are linking static libraries only if someone referenced any symbols defined in it, so we skip libexp (no one referenced ‘exp’).

We see dynamic linking of libm. We will link to it unconditionally, so linker captures all symbols in libm. It found definition of ‘exp’ and insert it to the linker symbol table.

We run LTO stage. /tmp/lto-llvm-8f9b9a.o was added to the linker inputs.

2nd pass:

There is a reference to exp in /tmp/lto-llvm-8f9b9a.o, but we have it’s definition already (in libm).

We see libexp again, but it’s too late, we already have exp defined in libm, and it won’t be overridden.

I confirmed all the above by adding debug prints in LLVMGold plugin (LLVMgold.so) and built debug version of binutils (added debug prints to gold linker).

Behavior is similar for gold and bfd linkers.

This problem goes away if I add ‘-u exp’ to the linker command. This option adds undefined symbol ‘exp’ to the linker symbol right at the beginning of the process, so it will be taken in 1st pass from libexp. But of course, it can’t be a solution to this issue.

How it affects us?

We have our own runtime library which provides implementation for math functions like exp, log, etc. So, we use it as a replacement for libm.

We link libm dynamically because not every distribution is supplied with static version of libm.

It’s not so trivial to report symbols that correspond to llvm intrinsics because we don’t know for sure to which exact symbols they will be expanded. For example, in rL322087 we started lowering llvm.exp.* to “__exp_finite” if –ffast-math (or -Ofast) is provided.

Any thoughts on this?

-Denis Bakhvalov

Employee of Intel, compiler development team.

A couple questions/notes so I can understand better:

Without -flto, a.o ends up with a reference to __exp_finite, which also would not be satifisfied out of libexp.a. Do you also have an implementation of __exp_finite in your libexp.a?

Can you build with -fno-builtin, or -fno-builtin-exp etc? That results in a reference to __exp_finite in the .o bitcode (which of course has the same issue I mentioned above, but is consistent). That seems to be the option you should use here when you want to use your own implementations.

Teresa

Hello Teresa,

Without -flto, a.o ends up with a reference to __exp_finite,

That’s correct.

which also would not be satifisfied out of libexp.a.

That’s not correct. Even if libexp.a would have __exp_finite, it wouldn’t be resolved from libexp.a, because of the behavior described in my first message.

Do you also have an implementation of __exp_finite in your libexp.a?

No, I don’t have __exp_finite in libexp.a. I presented complete code in my first message. You can reproduce it yourself on current trunk.

Can you build with -fno-builtin, or -fno-builtin-exp etc?

That results in a reference to __exp_finite in the .o bitcode

(which of course has the same issue I mentioned above, but is consistent).

That seems to be the option you should use here when you want to use your own implementations.

Yes, -fno-builtin results in call to __exp_finite in the bitcode. But the problem I try to address is not what version of exp libcall we want to generate: exp or __exp_finite.

The problem is that if the exp call was lowered to llvm intrinsic, linker doesn’t resolve it from my static library in LTO build, because plugin (LLVMgold.so) doesn’t report those symbols (yet in intrinsic form) to the linker.

Best regards,
Denis Bakhvalov.

+pcc for thoughts

I’m trying to understand how the non-LTO case is supposed to work
non-LTO case works because when linker starts it’s job all the llvm intrinsics are already lowered.

Right, which is why I am suggesting that it might be appropriate to build with -fno-builtin (or -fno-builtin-exp) here –

this solves the LTO issue as there will no longer be an llvm intrinsic in the bitcode

Yes, I just tested it and -fno-builtin helps in this case. I added definition of __exp_finite to libexp.a and now it was resolved from libexp.a (because no llvm intrinsics).

Best regards,
Denis Bakhvalov.

> <>I'm trying to understand how the non-LTO case is supposed to work
non-LTO case works because when linker starts it’s job all the llvm intrinsics are already lowered.

Yup, this has been a long-time problem for LTO. The right answer is for LTO to determine a fairly tight superset of the symbols that will be used after lowering, and request them from the linker (i.e., add them to the reported symbol table as undefined). This isn't particularly easy to do and there are workarounds, so no one has prioritized it.