LLJIT vs. thread-local storage (again)

Hi JITters,

I have some JIT-compiled C++ code that uses symbols from a DSO which
uses C++14's thread support library. When I compile it I get the
following error message:

Symbols not found: [ __emutls_v._ZSt15__once_callable,
__emutls_v._ZSt11__once_call ]

Those seem to correspond to std::__once_callable and std::__once_call,
which are indeed present in the C++ standard library. And I imagine
they are needed because the DSO makes calls to std::call_once(). But
ORC seems to be looking for them with the prefix "__emutls_v".

I had a similar problem in the past, which I thought Lang had fixed...
See this thread:

http://lists.llvm.org/pipermail/llvm-dev/2020-January/138402.html

Perhaps it wasn't as fixed as I thought? Or maybe there is something I
need to do for JIT to fully support thread-local storage?

Geoff

Hi Geoff,

If you want to access the variable directly from JIT’d code you will need to compile the DSO with -femulated-tls. Alternatively you could introduce a wrapper function to return the variable’s address.

This doesn’t help you right now, but for what it is worth the road-map to native TLS is becoming clearer:

(1) Land the orcv1-removal branch.
(2) Land the ORC remote-target-support branch – This breaks ORC up into three libraries: One for the JIT, one for the target process and one of shared data structures.
(3) Create an ORC runtime library. I’m not sure whether this should live in LLVM or compiler-rt, but it would include implementations of, among other things, native TLS variable setup.
(4) Teach JITLink to handle native TLS relocations. This should be relatively straightforward on MachO. It may be more involved for ELF, but I haven’t had a chance to dig in to this yet.

Steps (1) and (2) are mostly done. Step (3) shouldn’t be too difficult. Step (4) will be the interesting part.

– Lang.

Hmm, I'm confused. The DSO is compiled with gcc. Do I need to compile
it with clang instead? I don't believe the JITted code uses the thread
support library directly, although I suppose it may be hidden with
templates and/or inline functions...

Hi Geoff,

Sorry – I misread your question!

I have some JIT-compiled C++ code that uses symbols from a DSO which uses C++14’s thread support library. When I compile it I get the following error message:
Symbols not found: [ __emutls_v._ZSt15__once_callable,
__emutls_v._ZSt11__once_call ]

That error message implies that your JIT’d code is directly referencing a TLS variable, but the definition of the TLS variable has not been compiled with -femulated-tls.

Did you expect your JIT’d code to reference call_once? Or was this perhaps due to some unexpected inlining?

If JIT’d code doesn’t access TLS variables at all then you don’t have to worry about -femulated-tls. If JIT’d code uses TLS variables (either defining or referencing them) then you need to turn -femulted-tls on for the JIT’d code and any libraries that define TLS variables used by JIT’d code. Unfortunately this can snowball: Once the library is complied with emulated-tls, if it depends on TLS variables in a second library then that also needs to be compiled with -femulated-tls, and so on.

I had a similar problem in the past, which I thought Lang had fixed…

That was a different emulated-tls problem: The JIT tracks symbol definitions very carefully to ensure that they’re not dropped (or error out if they are) and the implicit renaming used by -femulated-tls was breaking this. The fix taught the JIT to anticipate the renaming in order to fix the tracking. This allows -femulated-tls to be used in JIT’d code. but you still need to use it consistently across library boundaries.

Native TLS should fix this whole mess once we can support it.

– Lang.

Hi Lang,

Yes, it turns out the JITted code accidentally references thread-local
stuff via a function template defined by the DSO which calls
std::call_once(). And I don't think compiling external libraries with
clang and -femulated-tls is a reasonable solution. So I guess we will
have to just wait until LLJIT supports native TLS...

Thanks for all your help,
Geoff