MCJIT

Hi All,

What is the current level of stability of MCJIT using ELF?

Thanks.

Hi Braxton,

I think it’s very good. All of the ExecutionEngine tests pass. I’m not aware of any major defects.

I don’t believe that the current implementation sets the permissions on JITed memory as you might wish. Namely, I think executable memory may be left writeable and data memory may be executable. The current interface is also somewhat limited. Basically, you pass a Module to the constructor, it JITs that Module immediately and then you can get the address of individual functions to execute them. What is there works though.

-Andy

Can you also tell about inline-asm support in MCJIT for x86? I've been trying to get it to work (my previous post: "JIT support for inline asm on Linux"), I get an error:

LLVM ERROR: Inline asm not supported by this streamer because we don't
have an asm parser for this target

thanks,
ashok

Inline asm should work in the MCJIT just as well as it does in statically compiled code. It's the same code path for both. The error you're seeing indicates that something isn't configured in a way the compiler understands.

-Jim

I was able to get past the error by calling InitializeNativeTargetAsmParser() in my code. Now I have a failure in resolving external libraries, so looking into that (recompiled with --enable-ffi but I now get an error LLVMgold.so not found).

Then I hda to disable the following code in lib/Target/X86/X86CodeEmitter.cpp:

      case TargetOpcode::INLINEASM:
        // We allow inline assembler nodes with empty bodies - they can
        // implicitly define registers, which is ok for JIT.
        //if (MI.getOperand(0).getSymbolName()[0])
        // report_fatal_error("in X86CodeEmitter.cpp JIT does not support inline asm!");
        break;

otherwise, when it sees a non-zero char in the inline asm (say load) and report failure and die.

Also, I also had to link in ${LLVM_TARGETS_TO_BUILD} into lli for it to compile after adding InitializeNativeTargetAsmParser() call in lli.cpp.

Can you point me how to resolve external symbols in my app (or how to enable it in lli)?

thanks,
ashok

I was able to get past the error by calling InitializeNativeTargetAsmParser() in my code. Now I have a failure in resolving external libraries, so looking into that (recompiled with --enable-ffi but I now get an error LLVMgold.so not found).

Then I hda to disable the following code in lib/Target/X86/X86CodeEmitter.cpp:

     case TargetOpcode::INLINEASM:
       // We allow inline assembler nodes with empty bodies - they can
       // implicitly define registers, which is ok for JIT.
       //if (MI.getOperand(0).getSymbolName()[0])
       // report_fatal_error("in X86CodeEmitter.cpp JIT does not support inline asm!");
       break;

otherwise, when it sees a non-zero char in the inline asm (say load) and report failure and die.

If you're hitting that code, you're running the old JIT (which does indeed not support inline assembly), not the MCJIT.

I was able to get past the error by calling InitializeNativeTargetAsmParser() in my code. Now I have a failure in resolving external libraries, so looking into that (recompiled with --enable-ffi but I now get an error LLVMgold.so not found).

Then I hda to disable the following code in lib/Target/X86/X86CodeEmitter.cpp:

      case TargetOpcode::INLINEASM:
        // We allow inline assembler nodes with empty bodies - they can
        // implicitly define registers, which is ok for JIT.
        //if (MI.getOperand(0).getSymbolName()[0])
        // report_fatal_error("in X86CodeEmitter.cpp JIT does not support inline asm!");
        break;

otherwise, when it sees a non-zero char in the inline asm (say load) and report failure and die.

If you're hitting that code, you're running the old JIT (which does indeed not support inline assembly), not the MCJIT.

Do I need to enable anything at configure, my configure looks like this:

../llvm/configure --enable-libffi --enable-targets=host-only --prefix=/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/bin

I added the enable-libffi when trying to figure out resolving external libs.

thanks,
ashok

No, selecting MCJIT vs. the old JIT is done in the EngineBuilder. If you're using lli, you can pass -use-mcjit on the command line. If you're using your own driver, you can use the lli.cpp for a reference.

-Jim

Thats what I was pointed to earlier, and following that I set the flag in EngineBuilder.setUseMCJIT(true). I looked at the code of EngineBuilder, it saves the flag but the flag doesnt seem to be used anywhere. Am I missing something?

thanks,
ashok

Hm. OK, that's odd. It should change which constructor gets called in EngineBuilder::create() (which is in lib/ExecutionEngine/ExecutionEngine.cpp). Are you perhaps calling setUseMCJIT(true) after having already called create()? Can you step through EngineBuilder::create() and see what's happening there?

-Jim

If you're hitting that code, you're running the old JIT (which does indeed not support inline assembly), not the MCJIT.

Do I need to enable anything at configure, my configure looks like this:

../llvm/configure --enable-libffi --enable-targets=host-only --prefix=/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/bin

I added the enable-libffi when trying to figure out resolving external libs.

No, selecting MCJIT vs. the old JIT is done in the EngineBuilder. If you're using lli, you can pass -use-mcjit on the command line. If you're using your own driver, you can use the lli.cpp for a reference.

-Jim

Thats what I was pointed to earlier, and following that I set the flag in EngineBuilder.setUseMCJIT(true). I looked at the code of EngineBuilder, it saves the flag but the flag doesnt seem to be used anywhere. Am I missing something?

Hm. OK, that's odd. It should change which constructor gets called in EngineBuilder::create() (which is in lib/ExecutionEngine/ExecutionEngine.cpp). Are you perhaps calling setUseMCJIT(true) after having already called create()? Can you step through EngineBuilder::create() and see what's happening there?

-Jim

lli.cpp instantiates the builder, sets options/flags etc and then calls create(). In my driver, I have:

llvm::EngineBuilder(m_LLVMMod).setErrorStr(&engErr).setUseMCJIT(true).create();

which is what you mentioned. I'll step through create() and check whats happening.

thanks,
ashok

Hm. OK, that's odd. It should change which constructor gets called in EngineBuilder::create() (which is in lib/ExecutionEngine/ExecutionEngine.cpp). Are you perhaps calling setUseMCJIT(true) after having already called create()? Can you step through EngineBuilder::create() and see what's happening there?

-Jim

lli.cpp instantiates the builder, sets options/flags etc and then calls
create(). In my driver, I have:

llvm::EngineBuilder(m_LLVMMod).setErrorStr(&engErr).setUseMCJIT(true).create();

which is what you mentioned. I'll step through create() and check whats
happening.

My bad about MCJIT flag not being used (must have the code before I switched to trunk). It *IS* used and the MCJIT constructor is being invoked.

So I reverted my changes, and recompiled and ran lli, but it gives the error:

LLVM ERROR: Inline asm not supported by this streamer because we don't have an asm parser for this target

I add back the call to InitializeNativeTargetAsmParser() in lli.cpp, but compilation fails:

make[4]: Leaving directory `/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/tools/clang/tools/arcmt-test'
/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/tools/lli/Debug+Asserts/lli.o: In function `llvm::InitializeNativeTargetAsmParser()':
/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/llvm/include/llvm/Support/TargetSelect.h:149: undefined reference to `LLVMInitializeX86AsmParser'
collect2: ld returned 1 exit status

So I add back to lli/CMakeLists.txt:
< set(LLVM_LINK_COMPONENTS mcjit jit interpreter nativecodegen bitreader asmparser selectiondag)

set(LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD} mcjit jit interpreter nativecodegen bitreader asmparser selectiondag)

lli/Makefile (a simple make invocation didnt seem to regenerate the Makefile):
<LINK_COMPONENTS := mcjit jit interpreter nativecodegen bitreader asmparser selectiondag

LINK_COMPONENTS := all-targets mcjit jit interpreter nativecodegen bitreader asmparser selectiondag

Compiles and runs till lli is unable to resolve an external symbol.

In short, tools/lli/[lli.cpp,CMakeLists.txt] will need to be modified for MCJIT to be used in lli. No need to comment out the code in lib/Target/X86/X86CodeEmitter.cpp.

Thanks for correcting me. Any hints on the external symbol resolution:

LLVM ERROR: Could not resolve external global address: _ZNSt3__14coutE

which seems to be cout from the std lib.

tia,
ashok

Hm. OK, that's odd. It should change which constructor gets called in EngineBuilder::create() (which is in lib/ExecutionEngine/ExecutionEngine.cpp). Are you perhaps calling setUseMCJIT(true) after having already called create()? Can you step through EngineBuilder::create() and see what's happening there?

-Jim

lli.cpp instantiates the builder, sets options/flags etc and then calls
create(). In my driver, I have:

llvm::EngineBuilder(m_LLVMMod).setErrorStr(&engErr).setUseMCJIT(true).create();

which is what you mentioned. I'll step through create() and check whats
happening.

My bad about MCJIT flag not being used (must have the code before I switched to trunk). It *IS* used and the MCJIT constructor is being invoked.

OK, cool. That's good news.

So I reverted my changes, and recompiled and ran lli, but it gives the error:

LLVM ERROR: Inline asm not supported by this streamer because we don't have an asm parser for this target

I add back the call to InitializeNativeTargetAsmParser() in lli.cpp, but compilation fails:

Makes sense.

make[4]: Leaving directory `/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/tools/clang/tools/arcmt-test'
/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/tools/lli/Debug+Asserts/lli.o: In function `llvm::InitializeNativeTargetAsmParser()':
/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/llvm/include/llvm/Support/TargetSelect.h:149: undefined reference to `LLVMInitializeX86AsmParser'
collect2: ld returned 1 exit status

So I add back to lli/CMakeLists.txt:
< set(LLVM_LINK_COMPONENTS mcjit jit interpreter nativecodegen bitreader asmparser selectiondag)

set(LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD} mcjit jit interpreter nativecodegen bitreader asmparser selectiondag)

lli/Makefile (a simple make invocation didnt seem to regenerate the Makefile):
<LINK_COMPONENTS := mcjit jit interpreter nativecodegen bitreader asmparser selectiondag

LINK_COMPONENTS := all-targets mcjit jit interpreter nativecodegen bitreader asmparser selectiondag

It seems odd to me that this is required, but I'm not an expert on LLVM's build system, so that doesn't mean much. Daniel, what do you think?

Compiles and runs till lli is unable to resolve an external symbol.

In short, tools/lli/[lli.cpp,CMakeLists.txt] will need to be modified for MCJIT to be used in lli. No need to comment out the code in lib/Target/X86/X86CodeEmitter.cpp.

Thanks for correcting me. Any hints on the external symbol resolution:

LLVM ERROR: Could not resolve external global address: _ZNSt3__14coutE

which seems to be cout from the std lib.

That's interesting. I would have expected a second leading underscore on that symbol. Perhaps not on your target? I suggest checking to see where that symbol (with or without the second '_') is defined and finding out which is expected. That'll give more information about whether this is failing because the symbol lookup isn't looking in the right places or because it's looking for the wrong thing.

-Jim

Thanks for correcting me. Any hints on the external symbol resolution:

LLVM ERROR: Could not resolve external global address: _ZNSt3__14coutE

which seems to be cout from the std lib.

That's interesting. I would have expected a second leading underscore on that symbol. Perhaps not on your target? I suggest checking to see where that symbol (with or without the second '_') is defined and finding out which is expected. That'll give more information about whether this is failing because the symbol lookup isn't looking in the right places or because it's looking for the wrong thing.

-Jim

Now I'm in a tricky situation. I recompiled LLVM/clang with --enable-libffi trying to find out how to resolve the external symbols. Now compiling with clang gives the following error:

ashoknn@neo-wk06:/local/mnt/workspace/ashoknn/crd/neo/llvm/proto$clang -emit-llvm testnoinline.c
/usr/bin/ld: /local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/bin/bin/../lib/LLVMgold.so: error loading plugin
/usr/bin/ld: /local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/bin/bin/../lib/LLVMgold.so: error in plugin cleanup (ignored)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
ashoknn@neo-wk06:/local/mnt/workspace/ashoknn/crd/neo/llvm/proto$

So I checked out a fresh copy of trunk, recompiled with:

../llvm/configure --enable-targets=host-only --prefix=/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmtip/build/bin

Notice I didnt specify libffi above, compilation still fails with the same error above. Any clue on how to get past this? I'm not worried about enabling LTO for now.

thanks,
ashok

Hm. OK, that's odd. It should change which constructor gets called in EngineBuilder::create() (which is in lib/ExecutionEngine/ExecutionEngine.cpp). Are you perhaps calling setUseMCJIT(true) after having already called create()? Can you step through EngineBuilder::create() and see what's happening there?

-Jim

lli.cpp instantiates the builder, sets options/flags etc and then calls
create(). In my driver, I have:

llvm::EngineBuilder(m_LLVMMod).setErrorStr(&engErr).setUseMCJIT(true).create();

which is what you mentioned. I'll step through create() and check whats
happening.

My bad about MCJIT flag not being used (must have the code before I switched to trunk). It *IS* used and the MCJIT constructor is being invoked.

OK, cool. That's good news.

So I reverted my changes, and recompiled and ran lli, but it gives the error:

LLVM ERROR: Inline asm not supported by this streamer because we don't have an asm parser for this target

I add back the call to InitializeNativeTargetAsmParser() in lli.cpp, but compilation fails:

Makes sense.

make[4]: Leaving directory `/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/tools/clang/tools/arcmt-test'
/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/build/tools/lli/Debug+Asserts/lli.o: In function `llvm::InitializeNativeTargetAsmParser()':
/local/mnt/workspace/ashoknn/crd/neo/llvm/proto/llvmsvn/llvm/include/llvm/Support/TargetSelect.h:149: undefined reference to `LLVMInitializeX86AsmParser'
collect2: ld returned 1 exit status

So I add back to lli/CMakeLists.txt:
< set(LLVM_LINK_COMPONENTS mcjit jit interpreter nativecodegen bitreader asmparser selectiondag)

set(LLVM_LINK_COMPONENTS ${LLVM_TARGETS_TO_BUILD} mcjit jit interpreter nativecodegen bitreader asmparser selectiondag)

lli/Makefile (a simple make invocation didnt seem to regenerate the Makefile):
<LINK_COMPONENTS := mcjit jit interpreter nativecodegen bitreader asmparser selectiondag

LINK_COMPONENTS := all-targets mcjit jit interpreter nativecodegen bitreader asmparser selectiondag

It seems odd to me that this is required, but I'm not an expert on LLVM's build system, so that doesn't mean much. Daniel, what do you think?

Right, this shouldn't be required. The nativecodegen component is the
one which is supposed to make sure that X86 gets linked in.

- Daniel