ORC global ctor handling

Beginners
1

I have a situation where global ctors in LLVM IR being sent to ORC JIT for compilation can’t be found by the JIT (on a lookup). The specific use case I have here is to save the object to a file (to be linked and executed later). Output with some debug info as well is below:

Looking up { (__mlir_load_ctor_1_kernel, RequiredSymbol) } in [ ("main", MatchAllSymbols) ] (required state: Ready)
Could not compile __mlir_load_ctor_1_kernel:
  Symbols not found: [ __mlir_load_ctor_1_kernel ]
Program aborted due to an unhandled Error:
Symbols not found: [ __mlir_load_ctor_1_kernel ]

This is what the LLVM IR looks like:

@llvm.global_ctors = appending global [3 x { i32, void ()*, i8* }] [{ i32, void ()*, i8* } { i32 0, void ()* @__mlir_load_ctor_1_kernel, i8* null }, ...]

define internal void @__mlir_load_ctor_1_kernel() !dbg !3 {
  %1 = call i8* ...
  ...
  ret void
}
...

I have no symbol lookup issues without the use of global ctors. Is it possible that the global ctor was somehow optimized away? I couldn’t immediately find a way to dump the IR right before the final code generation to confirm this, but any pointers to verify that or to ensure that the compilation succeeds will be appreciated. Note that I’m not trying to execute the IR yet and, thus, not expecting the initializers to be called on an invocation. I’m just looking up the list of functions in the module to dump the object for offline linking and execution. The same IR compiles, links, and executes fine if I use this route.

opt -O3 | llc -O3 --relocation-model=pic | as - -o aot.o
# Link aot.o and it succeeds; execute and the global ctor is correctly called.
...

UPDATE: After I had typed out the message above, I realized ORC JIT handles/replaces LLVM global ctors/dtors to generate a single “init” function: llvm-project/LLJIT.cpp at e4dd1d033063c8a669c7e77d9d7f88c1d3058e88 · llvm/llvm-project · GitHub
Not looking those up and dumping the compiled object:

$ nm orc.o | grep orc
00000000000009a0 T __orc_init_func.LLVMDialectModule

However, when I try to link and use the above object, I don’t see the initializer executing. Reading this elf, I find no init/ctors section:

$ readelf -SW orc.o  | grep init

However, the one generated via AOT:

$ readelf -SW aot.o  | grep init
  [ 6] .init_array.0     INIT_ARRAY      0000000000000000 004050 000018 08  WA  0   0  8
  [ 7] .rela.init_array.0 RELA            0000000000000000 004de0 000048 18   I 20   6  8

Is there a way one can get the ORC compiled object correctly have the static initializer?

Other relevant code:

CC: @lhames

2

Are you explicitly trying to load __mlir_load_ctor_1_kernel here? Or is this Orc figuring out by itself it should look it up?

The fact that this is an “internal” linkage function makes me think it may not be reached directly by name.

It’s not clear to me how you generated the object you’re looking at here: is this LLVM IR that flew through the JIT and you saved the temporary .o at the end?

3

The MLIR ExecutionEngine has a dumpToObjectFile to allow this: you can call it via the Python binding too (or via -dump-object-file with mlir-cpu-runner).

Explicitly, but it’s not my changes. The MLIR ExecutionEngine (existing code in tree) goes through all function names. Orc would also likely go through it when it processes the global ctors/dtors but the error above isn’t from that.

It’s not fully clear to me why those with internal linkage can’t be looked up by Orc. Anyway, as I commented later, they are processed and replaced by a single initializer, but for some reason (apparently due to missing client-side code), the static initializer isn’t being properly set up. However, with the other approach (opt + llc), the static initializer works as expected (even with the internal linkage).

4

Can you use LLJIT::initialize and LLJIT::deinitialize to run the initializers and deinitializers? These will try to run initializers and deinitializers in a way that’s compatible with the approach taken by the underlying Platform class.

In your case LLJIT is implicitly using the GenericIRPlatform support class that moves the initializers into the __orc_init_func function on all platforms, then searches for and runs that function when LLJIT::initialize is called. This works for LLVM IR in an object-format agnostic way, but will fail to find native initializers in object files added directly to the JIT (as opposed to being compiled from LLVM IR). It also does not include native initializers in objects produced by the JIT, as you have seen.

A better alternative, if you can build compiler-rt, is to use ExecutorNativePlatform and the ORC runtime to enable native initializers:

  LLJITBuilder()
    ...
    .setPlatformSetUp(ExecutorNativePlatform("/path/to/liborc_rt.a"));

The native platform classes (COFFPlatform, ELFNixPlatform and MachOPlatform) will all use the format’s native initializer representations.

1 Like
5

This is a side note to the main answer above, but ORC follows the usual linker rules for lookup: internal linkage symbols aren’t visible outside the file that defines them, so won’t show up in JITDylib level lookups.

For initializers the answer is, as described above, to ask LLJIT to run them for you. For general internal symbols that you want to access you can promote them to external linkage to make them visible.

1 Like
6

Thanks - I added this, and this part works as intended. I’m able to get the initializer running on the MLIR JIT path. I’ll submit this patch upstream.

Thank you. This answers my main question. As opposed to the JIT, getting the objects to have native initializers was the main thing I was after.

MLIRExecutionEngine (and I think most of MLIR) currently does not depend on compiler-rt; so this dependency on compiler-rt just for emitting objects with native initializers may not be ideal. Perhaps *Platform can be moved elsewhere. Is adding .setPlatformSetup all that’s needed to ensure the ORC generated object has native initializers? Thanks.

Okay. I did, however, notice that the generated object does have all the code for global ctors in spite of a lookup not being called on the ctors from the outside (I skipped them from the MLIR side).

nm orc_obj_dump.o | grep ctor
0000000000000000 t __mlir_load_ctor_1_kernel

So this part got compiled at the execution engine creation time and without an explicit lookup. (We do the explicit lookup to force compilation to do an object dump without having to execute, which is when it would have otherwise compiled it lazily).

7

They’re a package deal an the moment, with the design tied to the fact that ORC supports out-of-process execution of JIT’d code: The Platform classes scrape info out of the objects as they’re JIT-linked and send it (via the ExecutorProcessControl interface) to the ORC runtime to be executed. This allows JIT’d code to behave uniformly both in-process and out-of-process.

I would like to break the ORC runtime out of compiler-rt and into its own project in the future, and that may make it easier to pick up as a dependency. I don’t have a timeline for that project though.

To deal with the fact that initializer sections don’t typically have names associated with them (the initializer functions do, but the pointers to those functions are usually anonymous) ORC has a concept called “initializer symbols”. When a module containing an initializer is added we create a unique initializer symbol for it. When you call LLJIT::initialize it will implicitly lookup any registered initializer symbols, and that will trigger the emission of any modules containing global ctors.

1 Like