ORC Assertion failure

Hi

On Windows 10 when using a debug build of LLVM 10, I get this assertion failure:

Assertion failed: (KV.second.getFlags() & ~WeakFlags) == (I->second &
~WeakFlags) && "Resolving symbol with incorrect flags", file
C:\work\github\llvm-10.0.0.src\lib\ExecutionEngine\Orc\Core.cpp, line
450

The same failure occurred in LLVM 9 too:

Assertion failed: I->second == KV.second.getFlags() && "Resolving
symbol with incorrect flags", file
C:\work\github\llvm-9.0.0.src\lib\ExecutionEngine\Orc\Core.cpp, line
384

I am unsure what this assertion means and how I can debug this.

The assertion seems to occur just after JIT compiling a function, when
my code is trying to get the compiled function's address using
something like this:

return ES->lookup({MainJD}, (*Mangle)(Name.str()));

Here is the call stack:

     libravi.dll!llvm::orc::MaterializationResponsibility::notifyResolved(const
llvm::DenseMap<llvm::orc::SymbolStringPtr,llvm::JITEvaluatedSymbol,llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>,llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr,llvm::JITEvaluatedSymbol>>
& Symbols) Line 449 C++
     libravi.dll!llvm::orc::RTDyldObjectLinkingLayer::onObjLoad(unsigned
__int64 K, llvm::orc::MaterializationResponsibility & R,
llvm::object::ObjectFile & Obj,
std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>
LoadedObjInfo, std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>> Resolved,
std::set<llvm::StringRef,std::less<llvm::StringRef>,std::allocator<llvm::StringRef>>
& InternalSymbols) Line 232 C++

   libravi.dll!llvm::orc::RTDyldObjectLinkingLayer::emit::__l2::<lambda>(std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>> LoadedObjInfo, std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef const ,llvm::JITEvaluatedSymbol>>> ResolvedSymbols) Line 149 C++

     libravi.dll!llvm::unique_function<llvm::Error
__cdecl(std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>,std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>>)>::CallImpl<llvm::Error
<lambda>(std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>,
std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>>)>(void * CallableAddr,
std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>
& <Params_0>, std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>> & <Params_1>) Line 157 C++
     libravi.dll!llvm::unique_function<llvm::Error
__cdecl(std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>,std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>>)>::operator()(std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>
<Params_0>, std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>> <Params_1>) Line 280 C++
     libravi.dll!llvm::jitLinkForORC(llvm::object::ObjectFile & Obj,
std::unique_ptr<llvm::MemoryBuffer,std::default_delete<llvm::MemoryBuffer>>
UnderlyingBuffer, llvm::RuntimeDyld::MemoryManager & MemMgr,
llvm::JITSymbolResolver & Resolver, bool ProcessAllSections,
llvm::unique_function<llvm::Error
__cdecl(std::unique_ptr<llvm::RuntimeDyld::LoadedObjectInfo,std::default_delete<llvm::RuntimeDyld::LoadedObjectInfo>>,std::map<llvm::StringRef,llvm::JITEvaluatedSymbol,std::less<llvm::StringRef>,std::allocator<std::pair<llvm::StringRef
const ,llvm::JITEvaluatedSymbol>>>)> OnLoaded,
llvm::unique_function<void __cdecl(llvm::Error)> OnEmitted) Line 1427
  C++
     libravi.dll!llvm::orc::RTDyldObjectLinkingLayer::emit(llvm::orc::MaterializationResponsibility
R, std::unique_ptr<llvm::MemoryBuffer,std::default_delete<llvm::MemoryBuffer>>
O) Line 155 C++
     libravi.dll!llvm::orc::IRCompileLayer::emit(llvm::orc::MaterializationResponsibility
R, llvm::orc::ThreadSafeModule TSM) Line 41 C++
     libravi.dll!llvm::orc::IRTransformLayer::emit(llvm::orc::MaterializationResponsibility
R, llvm::orc::ThreadSafeModule TSM) Line 25 C++
     libravi.dll!llvm::orc::BasicIRLayerMaterializationUnit::materialize(llvm::orc::MaterializationResponsibility
R) Line 132 C++
     libravi.dll!llvm::orc::MaterializationUnit::doMaterialize(llvm::orc::JITDylib
& JD) Line 570 C++
     libravi.dll!llvm::orc::ExecutionSession::materializeOnCurrentThread(llvm::orc::JITDylib
& JD, std::unique_ptr<llvm::orc::MaterializationUnit,std::default_delete<llvm::orc::MaterializationUnit>>
MU) Line 1218 C++
     [External Code]
     libravi.dll!llvm::orc::ExecutionSession::dispatchMaterialization(llvm::orc::JITDylib
& JD, std::unique_ptr<llvm::orc::MaterializationUnit,std::default_delete<llvm::orc::MaterializationUnit>>
MU) Line 1204 C++
     libravi.dll!llvm::orc::ExecutionSession::runOutstandingMUs() Line
2188 C++
     libravi.dll!llvm::orc::ExecutionSession::lookup(llvm::orc::LookupKind
K, const std::vector<std::pair<llvm::orc::JITDylib *,enum
llvm::orc::JITDylibLookupFlags>,std::allocator<std::pair<llvm::orc::JITDylib
*,enum llvm::orc::JITDylibLookupFlags>>> & SearchOrder,
llvm::orc::SymbolLookupSet Symbols, llvm::orc::SymbolState
RequiredState, llvm::unique_function<void
__cdecl(llvm::Expected<llvm::DenseMap<llvm::orc::SymbolStringPtr,llvm::JITEvaluatedSymbol,llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>,llvm::detail::DenseMapPair<llvm::orc::SymbolStringPtr,llvm::JITEvaluatedSymbol>>>)>
NotifyComplete, std::function<void
__cdecl(llvm::DenseMap<llvm::orc::JITDylib
*,llvm::DenseSet<llvm::orc::SymbolStringPtr,llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>>,llvm::DenseMapInfo<llvm::orc::JITDylib
*>,llvm::detail::DenseMapPair<llvm::orc::JITDylib
*,llvm::DenseSet<llvm::orc::SymbolStringPtr,llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>>>>
const &)> RegisterDependencies) Line 2084 C++
     libravi.dll!llvm::orc::ExecutionSession::lookup(const
std::vector<std::pair<llvm::orc::JITDylib *,enum
llvm::orc::JITDylibLookupFlags>,std::allocator<std::pair<llvm::orc::JITDylib
*,enum llvm::orc::JITDylibLookupFlags>>> & SearchOrder, const
llvm::orc::SymbolLookupSet & Symbols, llvm::orc::LookupKind K,
llvm::orc::SymbolState RequiredState, std::function<void
__cdecl(llvm::DenseMap<llvm::orc::JITDylib
*,llvm::DenseSet<llvm::orc::SymbolStringPtr,llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>>,llvm::DenseMapInfo<llvm::orc::JITDylib
*>,llvm::detail::DenseMapPair<llvm::orc::JITDylib
*,llvm::DenseSet<llvm::orc::SymbolStringPtr,llvm::DenseMapInfo<llvm::orc::SymbolStringPtr>>>>
const &)> RegisterDependencies) Line 2124 C++
     libravi.dll!llvm::orc::ExecutionSession::lookup(const
std::vector<std::pair<llvm::orc::JITDylib *,enum
llvm::orc::JITDylibLookupFlags>,std::allocator<std::pair<llvm::orc::JITDylib
*,enum llvm::orc::JITDylibLookupFlags>>> & SearchOrder,
llvm::orc::SymbolStringPtr Name) Line 2145 C++
     libravi.dll!llvm::orc::ExecutionSession::lookup(llvm::ArrayRef<llvm::orc::JITDylib
*> SearchOrder, llvm::orc::SymbolStringPtr Name) Line 2157 C++
     libravi.dll!ravi::RaviJITState::findSymbol(llvm::StringRef Name)
Line 523 C++

Thanks and Regards
Dibyendu

Hi,

Some more info:

The values that are causing the failure are:

I.second {TargetFlags=0 '\0' Flags=None (0 '\0') }
KV.second {Address=1438723211264 Flags={TargetFlags=0 '\0'
Flags=Callable (32 ' ') } }

In line:

    assert((KV.second.getFlags() & ~WeakFlags) == (I->second & ~WeakFlags) &&
           "Resolving symbol with incorrect flags");

Interestingly this assertion failure occurs with ORC v2 but not ORC v1.

Regards

Hi Dibyendu,

Are you using LLJIT?

If you’re using a custom JIT class you will need to call ‘ObjLinkingLayer->setOverrideObjectFlagsWithResponsibilityFlags(true);’ when running on Windows, as RuntimeDyldCOFF does not correctly report symbol flags yet.

– Lang.

Hi Lang,

Thanks for looking at this.

Are you using LLJIT?

No.

If you're using a custom JIT class you will need to call 'ObjLinkingLayer->setOverrideObjectFlagsWithResponsibilityFlags(true);' when running on Windows, as RuntimeDyldCOFF does not correctly report symbol flags yet.

I tried this but I still get the assertion failure.

Hi Dibyendu,

Thanks for looking at this.

No worries!

If you’re using a custom JIT class you will need to call ‘ObjLinkingLayer->setOverrideObjectFlagsWithResponsibilityFlags(true);’ when running on Windows, as RuntimeDyldCOFF does not correctly report symbol flags yet.

I tried this but I still get the assertion failure.

The values that are causing the failure are:

I.second {TargetFlags=0 ‘\0’ Flags=None (0 ‘\0’) }
KV.second {Address=1438723211264 Flags={TargetFlags=0 ‘\0’
Flags=Callable (32 ’ ') } }

Very interesting. If you look at that assertion in context (see: https://github.com/llvm/llvm-project/blob/b11ecd196540d87cb7db190d405056984740d2ce/llvm/lib/ExecutionEngine/Orc/Core.cpp#L449) you will see that the I variable refers to the table entry for the MaterializationResponsibility, and KV to the table entry of the Symbols argument that is supplied to notifyResolve. That means that this symbol is being initially defined as non-callable, but then resolved as if it were callable.

Are you able to share the module that is causing this?

If you are building LLVM from source, are you able to add some debug logging to MaterializationResponsibility::notifyResolved to dump the offending symbol’s name, *KV.first?

My next step after that would be to determine whether this symbol should have been marked callable up-front (in which case the bug was probably at MaterializationUnit construction time) or whether it was mistakenly marked as callable at resolution time by RTDyldObjectLinkingLayer. I would also want to know why OverrideObjectFlags failed to update the flags here: https://github.com/llvm/llvm-project/blob/b11ecd196540d87cb7db190d405056984740d2ce/llvm/lib/ExecutionEngine/Orc/RTDyldObjectLinkingLayer.cpp#L213

Regards,
Lang.

Hi Dibyendu,

Have you fixed this issue? I run into the same issue as you…

Lin

| lin.lee
February 22 |

  • | - |

Hi Dibyendu,

Have you fixed this issue? I run into the same issue as you…

Hi Lin

I moved off LLVM so have not looked at this further.

Regards
Dibyendu

Got it. Thanks for the feedback.