Fail to load a pointer to a function inside MCJIT-ed code when it is reload from ObjectCache

Hi, All

I have a problem to reuse mcjit jitted code loaded from ObjectCache from a file. In the first run, I use MCJIT generate function JittedOpExpr object code as following and it runs OK. 0x7fe4801fa1f8 at instruction 0x00007fe4cc6c2014 points to 0x69382E which is the beginning of ExecEvalVar function. Then I save the object code into a file after implementing notifyObjectCompiled method.

IrExprGetValue:
0x00007fe4cc6c2000: push %rbp
0x00007fe4cc6c2001: mov %rsp,%rbp
0x00007fe4cc6c2004: mov 0x10(%rdi),%rax
0x00007fe4cc6c2008: pop %rbp
0x00007fe4cc6c2009: jmpq *%rax
0x00007fe4cc6c200b: nopl 0x0(%rax,%rax,1)
JittedOpExpr:
0x00007fe4cc6c2010: push %rbp
0x00007fe4cc6c2011: mov %rsp,%rbp
0x00007fe4cc6c2014: movabs $0x7fe4801fa1f8,%rax
0x00007fe4cc6c201e: movabs $0x7fe4801fa1e8,%rdi
0x00007fe4cc6c2028: callq *(%rax)
0x00007fe4cc6c202a: add $0x5,%rax
0x00007fe4cc6c202e: pop %rbp
0x00007fe4cc6c202f: retq
0x00007fe4cc6c2030: adc $0x0,%al
0x00007fe4cc6c2032: add %al,(%rax)
0x00007fe4cc6c2034: add %al,(%rax)

0x7fe4801fa1f8 → 0x69382E
ExecEvalVar(ExprState*, ExprContext*, bool*, ExprDoneCond*):
0x000000000069382e: push %rbp
0x000000000069382f: mov %rsp,%rbp
0x0000000000693832: push %r12

In the next run, I buildedunction of JittedOpExpr again and loaded compiled object from that saved binary file using getObject and went through getPointertoFunction from MCJIT execution engine, I got the following object code in memory. But this time 0x7fe4801fa1f8 points to 0x0, so when callq *(%rax) it couldn’t find the ExecEvalVar anymore. I followed the blog “Object Caching with the Kaleidoscope Example Problem” written by Andy Kaylor by implementing my own MCJITObjectCache class. Did I miss anything here? Thank you very much.

IrExprGetValue:
0x00007fe4cc6c2000: push %rbp
0x00007fe4cc6c2001: mov %rsp,%rbp
0x00007fe4cc6c2004: mov 0x10(%rdi),%rax
0x00007fe4cc6c2008: pop %rbp
0x00007fe4cc6c2009: jmpq *%rax
0x00007fe4cc6c200b: nopl 0x0(%rax,%rax,1)
JittedOpExpr:
0x00007fe4cc6c2010: push %rbp
0x00007fe4cc6c2011: mov %rsp,%rbp
0x00007fe4cc6c2014: movabs $0x7fe4801fa1f8,%rax
0x00007fe4cc6c201e: movabs $0x7fe4801fa1e8,%rdi
0x00007fe4cc6c2028: callq *(%rax)
0x00007fe4cc6c202a: add $0x5,%rax
0x00007fe4cc6c202e: pop %rbp
0x00007fe4cc6c202f: retq

but 0x7fe4801fa1f8 → 000000000

Hi Cheng,

It sounds like the either (1) 0x7fe4801fa1f8 is part of the environment, and it’s not being configured the same way the 2nd time around, or (2) The JIT is handling on-disk objects differently from in-memory ones. The first option is more likely.

It would be helpful if you could attach the object that was stored in your cache.

Cheers,
Lang.

Thank you Lang. I attached the ELF object file here for your reference. Here is the IR dump of JittedOpExpr LLVM function. IrExprGetValue1 LLVM function calls to external function expr->evalfunc(expr, econtext, isNull, isDone); which should be pointed by 0x7fe4801fa1f8. However, only the first time MCJIT generated object point to expr->evalfunc but second time when program load from object cache does not. In the second time, before program load object cache, brand new llvm::Module and llvm::ExecutionEngine are created. IrExprGetValue is saved in file as IR format and will be loaded into module every run. JittedOpExpr and other LLVM function are created and associated with the new Module.

; Function Attrs: uwtable
define i64 @IrExprGetValue1(%struct.ExprState* %expr, %struct.ExprContext* %econtext, i8* %isNull, i32* %isDone) #0 {
entry:
%expr.addr = alloca %struct.ExprState*, align 8
%econtext.addr = alloca %struct.ExprContext*, align 8
%isNull.addr = alloca i8*, align 8
%isDone.addr = alloca i32*, align 8
store %struct.ExprState* %expr, %struct.ExprState** %expr.addr, align 8
store %struct.ExprContext* %econtext, %struct.ExprContext** %econtext.addr, align 8
store i8* %isNull, i8** %isNull.addr, align 8
store i32* %isDone, i32** %isDone.addr, align 8
%0 = load %struct.ExprState** %expr.addr, align 8
%evalfunc = getelementptr inbounds %struct.ExprState* %0, i32 0, i32 2
%evalfunc1 = bitcast {}** %evalfunc to i64 (%struct.ExprState*, %struct.ExprContext*, i8*, i32*)**
%1 = load i64 (%struct.ExprState*, %struct.ExprContext*, i8*, i32*)** %evalfunc1, align 8
%2 = load %struct.ExprState** %expr.addr, align 8
%3 = load %struct.ExprContext** %econtext.addr, align 8
%4 = load i8** %isNull.addr, align 8
%5 = load i32** %isDone.addr, align 8
%call = call i64 %1(%struct.ExprState* %2, %struct.ExprContext* %3, i8* %4, i32* %5)
ret i64 %call
}

define i64 @JittedIntLit() {
entry:
ret i64 5
}

define i64 @JittedOpExpr(%struct.ExprState* %expr, %struct.ExprContext* %econtext, i8* %isNull, i32* %isDone) {
entry:
%lhs = call i64 @IrExprGetValue1(%struct.ExprState* inttoptr (i64 140715076067816 to %struct.ExprState*), %struct.ExprContext* %econtext, i8* %isNull, i32* %isDone)
%rhs = call i64 @JittedIntLit()
%tmp_add = add i64 %lhs, %rhs
ret i64 %tmp_add
}

and /samba/data/gDB2/src/ptcompiler/compiler/llvm_ir/GaussDB.ir contains

extern “C”
Datum IrExprGetValue(ExprState* expr, ExprContext *econtext, bool *isNull, ExprDoneCond *isDone) {
return expr->evalfunc(expr, econtext, isNull, isDone);
}

Cheng

jitted_int_add (1.24 KB)

Hi Cheng,

Thanks for attaching the object file and bit code - it’s very helpful.

I think the problem is that there is a pointer address hard-coded into your IR. If the data that pointer references is stored at a different address on your second run (which is highly likely) you get the failure that you’re seeing.

In your definition of JittedOpExpr, the first call instruction (to IRExprGetValue1) is:

%lhs = call i64 @IrExprGetValue1(%struct.ExprState* inttoptr (i64 140715076067816 to %struct.ExprState*), %struct.ExprContext* %econtext, i8* %isNull, i32* %isDone)

Note the hard coded pointer address 140715076067816 (0x7ffac81fa1e8). That should probably have been a reference to the %expr argument of JittedOpExpr. If you change this call to use %expr rather than the hard-coded address I think it will fix your issue.

I hope this helps.

Regards,
Lang.

Thank you Lang. You are right. The structure ExprState is allocated in every run, I should not hard code the pointer in IR. I tried to google an example using IRBuilder to allocate structure and access its element, but couldn’t find any. Could you give me a simple example on it? Once I create a new ExprState and pass it to @IrExprGetValue1 I think the problem should be solved.

Hi Cheng,

From the snippet of IR that you attached, it looks like your IR already has a definition for the ExprState type.

Operating on the assumption that you are calling JittedOpExpr yourself via MCJIT->getSymbolAddress() (Or one of the runFunction methods). In that case I think there are two things you do:

(1) When you are building the IR for JittedOpExpr you need to make sure that the call to IrExprGetValue1 references the %expr argument of JittedOpExpr, rather than a hard-coded pointer.

(2) In your JIT, you will need to allocate an ExprState object. This could just be done on the stack. The code would be along the lines of:

Function* F = M->getFunction(“JittedOpExpr”);
ExprState S;
JIT->runFunction(F, { GenericValue(&S); });

Apologies in advance if I’ve misunderstood your use-case. If the above does not work for you, could you please file a bug at llvm.org/bugs and attach the full JIT’ed IR and JIT driver code?

Cheers,
Lang.