LLVM 10 ORC2 issue with symbol resolution

Hi,

I updated my project to LLVM 10.0 today and I am getting JIT symbol
resolution errors.
I could not find any example or updated tutorial or documentation that
describes the new api - as all documentation seems out of date.

I paste below some code snippets that show what I am doing:

/* global syms is a array mapping names to function addresses */

ES->createJITDylib("<main>");
MainJD = ES->getJITDylibByName("<main>");
MainJD->addGenerator(
    cantFail(llvm::orc::DynamicLibrarySearchGenerator::GetForCurrentProcess(
        DL->getGlobalPrefix())));
auto &JD = *MainJD;
llvm::orc::MangleAndInterner mangle(*ES, *this->DL);
llvm::orc::SymbolMap Symbols;
for (int i = 0; global_syms[i].name != nullptr; i++) {
  Symbols.insert({mangle(global_syms[i].name),

llvm::JITEvaluatedSymbol(llvm::pointerToJITTargetAddress(global_syms[i].address),

llvm::JITSymbolFlags(llvm::JITSymbolFlags::FlagNames::Absolute))});
}
llvm::cantFail(JD.define(llvm::orc::absoluteSymbols(Symbols)), "Failed
to install extern symbols");

FYI - my previous post on similar issue.
Basically I haven't been able to get ORC v2 working - tried in 8, 9 and now 10.

http://llvm.1065342.n5.nabble.com/llvm-dev-ORC-v2-question-tp130489p130601.html

Thanks and Regards
Dibyendu

Hi Dibyendu,

I updated my project to LLVM 10.0 today and I am getting JIT symbol resolution errors.
I could not find any example or updated tutorial or documentation that describes the new api - as all documentation seems out of date.

The doxygen comments should be up-to-date, as should http://llvm.org/docs/ORCv2.html. You can find up-to-date example code for LLJIT in llvm/examples/LLJITExamples (LLVM 10) or llvm/examples/OrcV2Examples (master). If you come across any documentation in master that appears to be out of date please let me know and I will fix it.

The BuildingAJIT tutorial series is known to be out of date. I’m hoping to have that fixed by LLVM 11 but there is some feature work that I need to complete first.

Can you share the specific errors that you’re getting back? I would recommend replacing

cantFail(callToFailingFunc(…));

with

logAllUnhandledErrors(callToFailingFunc(…), errs(), “callToFailingFunc(…) failed:”);

Regards,
Lang.

Hi Lang,

I updated my project to LLVM 10.0 today and I am getting JIT symbol resolution errors.
I could not find any example or updated tutorial or documentation that describes the new api - as all documentation seems out of date.

The doxygen comments should be up-to-date, as should http://llvm.org/docs/ORCv2.html. You can find up-to-date example code for LLJIT in llvm/examples/LLJITExamples (LLVM 10) or llvm/examples/OrcV2Examples (master). If you come across any documentation in master that appears to be out of date please let me know and I will fix it.

Sadly http://llvm.org/docs/ORCv2.html is out of date. For example:

auto &JD = ES.getMainJITDylib();

This api no longer exists.

I did look at OrcV2Examples but did not see any that were trying to
set up external symbols. Did I miss something?

The BuildingAJIT tutorial series is known to be out of date. I'm hoping to have that fixed by LLVM 11 but there is some feature work that I need to complete first.

Can you share the specific errors that you're getting back? I would recommend replacing

cantFail(callToFailingFunc(...));

with

logAllUnhandledErrors(callToFailingFunc(...), errs(), "callToFailingFunc(...) failed:");

Okay thank you - I will do that and report back.

Hi Lang,

> Can you share the specific errors that you're getting back? I would recommend replacing
>
> cantFail(callToFailingFunc(...));
>
> with
>
> logAllUnhandledErrors(callToFailingFunc(...), errs(), "callToFailingFunc(...) failed:");
>

Okay thank you - I will do that and report back.

I had a look - to be honest I am not sure where the error is being
reported from. It looks like this:

JIT session error: Symbols not found: [ luaV_tointeger_, luaG_runerror ]

I don't think it is from any logging I am doing.

Hi Dibyendu,

I had a look - to be honest I am not sure where the error is being
reported from. It looks like this:
JIT session error: Symbols not found: [ luaV_tointeger_, luaG_runerror ]
I don’t think it is from any logging I am doing.

Errors in ORC may trigger cascading failures. E.g. if two different modules M1 and M2 both reference a function “foo” which cannot be resolved, then you have three errors: A failure to resolve “foo”, A failure to materialize the symbols in M1, and a failure to materialize the symbols in M2. When such a cascade occurs ORC will usually report the original error(s) via ExecutionSession::reportError (which logs them to stderr by default) and return a derived error (e.g. failure-to-materialize) to callers. I believe you’re seeing the logged error, and you will see a failure-to-materialize error if you log the result that reaches cantFail in your example above.

– Lang.

Hi Lang,

Hi Dibyendu,

I am not sure the --export-dynamic applies as I am explicitly
registering my symbols.
llvm::orc::MangleAndInterner mangle(*ES, *this->DL);
llvm::orc::SymbolMap Symbols;
for (int i = 0; global_syms[i].name != nullptr; i++) {
Symbols.insert({mangle(global_syms[i].name),

Ahh. You’re right: If you are managing registration yourself then process symbol visibility should not be an issue.

In this case the first thing that I would do to diagnose the problem is dump the content of your Symbols map and ensure that there really is an entry in there for the symbols that generated the missing-symbols error (luaV_tointeger_ and luaG_runerror).

llvm::JITSymbolFlags::FlagNames::Absolute

This can just be written as llvm::JITSymbolFlags::Absolute. If the JITDylib you are putting this in is different from the one that you’re adding JIT’d code to you will also need to make sure that the symbol is exported:

llvm::JITSymbolFlags Flags;
Flags |= llvm::JITSymbolFlags::Absolute;
Flags |= llvm::JITSymbolFlags::Exported;

Symbols that are not marked exported are hidden and not visible by default to code in other JITDylibs.

Regards,
Lang.

Hi Dibyendu,

I am a bit confused about this. I assumed that cantFail() will abort
the process on error - but my process doesn’t abort.
I have cantFail() in a few places, I am not sure if you are suggesting
adding the error logging everywhere …

Are you building LLVM in +Asserts mode? I believe that cantFail should abort in +Asserts builds, but it’s effectively a no-op if asserts are disabled.

If you are building in +Asserts mode it could also be consumeError: that function is intended to silently consume errors.

Both cantFail and consumeError should be used very carefully. The cantFail function should only be used if you are 100% certain that an error can never occur at runtime if program invariants hold. E.g. adding a symbol to a newly created JITDylib can’t fail because there are no existing definitions to clash with. The consumeError function should only be used if you have genuinely dealt with whatever error condition arose. If you think that an error is being silently dropped please see if you can trace it by putting a breakpoint on the ErrorInfoBase constructor and destructor, then following any errors generated up the stack to the point where they’re dropped. Silently dropped errors (in +Asserts mode) are likely bugs in the llvm::Error system and we definitely want to get them fixed.

Regards,
Lang.

Hi Lang,

Hi Dibyendu,

Thanks for the help.

No worries! Glad you were able to get it working. :slight_smile:

Regards,
Lang.