Orc JIT v2 breaks OpenMP in 11.x branch?

Greetings, Lang and other JITters,

Last week I moved our Orc v2-based app from top-of-tree to the new
11.x branch, and it no longer handles C++ code containing OpenMP
directives correctly.

More specifically, if I JIT compile a function containing OpenMP
parallel/for pragmas, using a release version of LLVM, the code
compiles and seems to work, but if I try to write the module's bitcode
to file, I get the following error message:

    Invalid record (Producer: 'LLVM11.0.0' Reader: 'LLVM 11.0.0').

If I use a debug version of LLVM, compilation fails with a broken assertion:

    .../llvm/lib/IR/Instructions.cpp:455: void
llvm::CallInst::init(llvm::FunctionType*, llvm::Va
lue*, llvm::ArrayRef<llvm::Value*>,
llvm::ArrayRef<llvm::OperandBundleDefT<llvm::Value*> >, const
llvm::Twine&): Assertion `(i >
= FTy->getNumParams() || FTy->getParamType(i) == Args[i]->getType())
&& "Calling a function with a bad signature!"' failed.

The same code compiles correctly using clang from the command line. It
also works fine using Orc v2 in the master branch of LLVM from the end
of May.

The problems are way too deep in the LLVM code for me to make any
sense of them... Does anybody else have a clue what might have
happened to Orc and/or OpenMP in the past couple months to trigger
this?

Geoff

Here, by the way, is the stack trace leading to the failed assertion.
Note #9: clang::CodeGen::CGOpenMPRuntime::emitForStaticFinish().

#0 0x00007fb70fa44870 in abort () from /usr/lib64/libc.so.6
#1 0x00007fb70fa3c0e6 in __assert_fail_base () from /usr/lib64/libc.so.6
#2 0x00007fb70fa3c192 in __assert_fail () from /usr/lib64/libc.so.6
#3 0x00007fb715ca1a68 in llvm::CallInst::init (this=0x51f6e10,
FTy=0x51fd280, Func=0x51f6d28, Args=..., Bundles=...,
    NameStr=...) at
/u/geoff/llvmrepo/llvm-project/llvm/lib/IR/Instructions.cpp:453
#4 0x00007fb7116a9e4e in llvm::CallInst::CallInst (this=0x51f6e10,
Ty=0x51fd280, Func=0x51f6d28, Args=..., Bundles=...,
    NameStr=..., InsertBefore=0x0) at
/u/geoff/llvmrepo/llvm-project/llvm/include/llvm/IR/Instructions.h:1681
#5 0x00007fb7116a9cf2 in llvm::CallInst::Create (Ty=0x51fd280,
Func=0x51f6d28, Args=..., Bundles=..., NameStr=...,
    InsertBefore=0x0) at
/u/geoff/llvmrepo/llvm-project/llvm/include/llvm/IR/Instructions.h:1489
#6 0x00007fb71171f1e0 in llvm::IRBuilderBase::CreateCall
(this=0x7fffd4fdaac8, FTy=0x51fd280, Callee=0x51f6d28, Args=...,
    OpBundles=..., Name=..., FPMathTag=0x0) at
/u/geoff/llvmrepo/llvm-project/llvm/include/llvm/IR/IRBuilder.h:2338
#7 0x00007fb71171f2d5 in llvm::IRBuilderBase::CreateCall
(this=0x7fffd4fdaac8, Callee=..., Args=..., OpBundles=..., Name=...,
    FPMathTag=0x0) at
/u/geoff/llvmrepo/llvm-project/llvm/include/llvm/IR/IRBuilder.h:2356
#8 0x00007fb7117124eb in
clang::CodeGen::CodeGenFunction::EmitRuntimeCall (this=0x7fffd4fda9d0,
callee=..., args=...,
    name=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGCall.cpp:4051
#9 0x00007fb7118b601f in
clang::CodeGen::CGOpenMPRuntime::emitForStaticFinish (this=0x4614460,
CGF=..., Loc=...,
    DKind=llvm::omp::OMPD_for) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:2875
#10 0x00007fb71197c294 in
clang::CodeGen::CodeGenFunction::<lambda(clang::CodeGen::CodeGenFunction&)>::operator()(clang::CodeGen::CodeGenFunction
&) const (__closure=0x7fffd4fd8340, CGF=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:3048
#11 0x00007fb71199754d in
llvm::function_ref<void(clang::CodeGen::CodeGenFunction&)>::callback_fn<clang::CodeGen::CodeGenFunction::EmitOMPWorksharingLoop(const
clang::OMPLoopDirective&, clang::Expr*, const CodeGenLoopBoundsTy&,
const CodeGenDispatchBoundsTy&)::<lambda(clang::CodeGen::CodeGenFunction&)>

(intptr_t, clang::CodeGen::CodeGenFunction &)

(callable=140736766772032,
    params#0=...) at
/u/geoff/llvmrepo/llvm-project/llvm/include/llvm/ADT/STLExtras.h:186
#12 0x00007fb71199ef91 in llvm::function_ref<void
(clang::CodeGen::CodeGenFunction&)>::operator()(clang::CodeGen::CodeGenFunction&)
const (this=0x7fffd4fd7b40, params#0=...) at
/u/geoff/llvmrepo/llvm-project/llvm/include/llvm/ADT/STLExtras.h:203
#13 0x00007fb71199d79a in
clang::CodeGen::CodeGenFunction::OpenMPCancelExitStack::emitExit(clang::CodeGen::CodeGenFunction&,
llvm::omp::Directive, llvm::function_ref<void
(clang::CodeGen::CodeGenFunction&)>) (this=0x7fffd4fdb678, CGF=...,
    Kind=llvm::omp::OMPD_for, CodeGen=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenFunction.h:1361
#14 0x00007fb71197d41e in
clang::CodeGen::CodeGenFunction::EmitOMPWorksharingLoop(clang::OMPLoopDirective
const&, clang::Expr*,
llvm::function_ref<std::pair<clang::CodeGen::LValue,
clang::CodeGen::LValue> (clang::CodeGen::CodeGenFunction&,
clang::OMPExecutableDirective const&)> const&,
llvm::function_ref<std::pair<llvm::Value*, llvm::Value*>
(clang::CodeGen::CodeGenFunction&, clang::OMPExecutableDirective
const&, clang::CodeGen::Address, clang::CodeGen::Address)> const&)
(this=0x7fffd4fda9d0, S=...,
    EUB=0x4cb7250, CodeGenLoopBounds=..., CGDispatchBounds=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:3050
#15 0x00007fb71197f746 in emitWorksharingDirective (CGF=..., S=...,
HasCancel=false)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:3335
#16 0x00007fb71197f7b0 in
clang::CodeGen::CodeGenFunction::<lambda(clang::CodeGen::CodeGenFunction&,
clang::CodeGen::PrePostActionTy&)>::operator()(clang::CodeGen::CodeGenFunction
&, clang::CodeGen::PrePostActionTy &) const (__closure=0x7fffd4fd8940,
    CGF=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:3346
#17 0x00007fb71199771f in
clang::CodeGen::RegionCodeGenTy::CallbackFn<clang::CodeGen::CodeGenFunction::EmitOMPForDirective(const
clang::OMPForDirective&)::<lambda(clang::CodeGen::CodeGenFunction&,
clang::CodeGen::PrePostActionTy&)> >(intptr_t,
clang::CodeGen::CodeGenFunction &, clang::CodeGen::PrePostActionTy &)
(CodeGen=140736766773568, CGF=..., Action=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.h:80
#18 0x00007fb7118a8787 in clang::CodeGen::RegionCodeGenTy::operator()
(this=0x4d7e948, CGF=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:595
#19 0x00007fb7118ab598 in (anonymous
namespace)::CGOpenMPRegionInfo::EmitBody (this=0x4d7e8d0, CGF=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:1041
#20 0x00007fb7118cce36 in
clang::CodeGen::CGOpenMPRuntime::emitInlinedDirective (this=0x4614460,
CGF=...,
    InnerKind=llvm::omp::OMPD_for, CodeGen=..., HasCancel=false)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:6222
#21 0x00007fb71197f8bd in
clang::CodeGen::CodeGenFunction::EmitOMPForDirective
(this=0x7fffd4fda9d0, S=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:3353
#22 0x00007fb711952aeb in clang::CodeGen::CodeGenFunction::EmitStmt
(this=0x7fffd4fda9d0, S=0x46c3b18, Attrs=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:202
#23 0x00007fb7119537d3 in
clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope
(this=0x7fffd4fda9d0, S=...,
    GetLast=false, AggSlot=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:453
#24 0x00007fb7119534b6 in
clang::CodeGen::CodeGenFunction::EmitCompoundStmt
(this=0x7fffd4fda9d0, S=..., GetLast=false,
    AggSlot=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:404
#25 0x00007fb7119532b8 in
clang::CodeGen::CodeGenFunction::EmitSimpleStmt (this=0x7fffd4fda9d0,
S=0x46c4750)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:377
#26 0x00007fb711952547 in clang::CodeGen::CodeGenFunction::EmitStmt
(this=0x7fffd4fda9d0, S=0x46c4750, Attrs=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:53
#27 0x00007fb7119757e9 in
clang::CodeGen::CodeGenFunction::<lambda(clang::CodeGen::CodeGenFunction&,
clang::CodeGen::PrePostActionTy&)>::operator()(clang::CodeGen::CodeGenFunction
&, clang::CodeGen::PrePostActionTy &) const (__closure=0x7fffd4fdc360,
    CGF=..., Action=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:1733
#28 0x00007fb711996ecf in
clang::CodeGen::RegionCodeGenTy::CallbackFn<clang::CodeGen::CodeGenFunction::EmitOMPParallelDirective(const
clang::OMPParallelDirective&)::<lambda(clang::CodeGen::CodeGenFunction&,
clang::CodeGen::PrePostActionTy&)> >(intptr_t,
clang::CodeGen::CodeGenFunction &, clang::CodeGen::PrePostActionTy &)
(CodeGen=140736766788448, CGF=..., Action=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.h:80
#29 0x00007fb7118a8787 in clang::CodeGen::RegionCodeGenTy::operator()
(this=0x7fffd4fda988, CGF=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:595
#30 0x00007fb7118ab598 in (anonymous
namespace)::CGOpenMPRegionInfo::EmitBody (this=0x7fffd4fda910,
CGF=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:1041
#31 0x00007fb71196e1e1 in
clang::CodeGen::CodeGenFunction::GenerateOpenMPCapturedStmtFunction
(this=0x7fffd4fda9d0, S=...,
    Loc=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:615
#32 0x00007fb7118ace11 in emitParallelOrTeamsOutlinedFunction
(CGM=..., D=..., CS=0x4a693a8, ThreadIDVar=0x49fe398,
    InnermostKind=llvm::omp::OMPD_parallel, OutlinedHelperName=..., CodeGen=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:1289
#33 0x00007fb7118acec7 in
clang::CodeGen::CGOpenMPRuntime::emitParallelOutlinedFunction
(this=0x4614460, D=...,
    ThreadIDVar=0x49fe398, InnermostKind=llvm::omp::OMPD_parallel, CodeGen=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGOpenMPRuntime.cpp:1297
#34 0x00007fb7119745c9 in emitCommonOMPParallelDirective (CGF=...,
S=..., InnermostKind=llvm::omp::OMPD_parallel, CodeGen=...,
    CodeGenBoundParameters=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:1530
#35 0x00007fb711975bd9 in
clang::CodeGen::CodeGenFunction::EmitOMPParallelDirective
(this=0x7fffd4fdc7e0, S=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmtOpenMP.cpp:1740
#36 0x00007fb711952aa5 in clang::CodeGen::CodeGenFunction::EmitStmt
(this=0x7fffd4fdc7e0, S=0x4a695a8, Attrs=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:196
#37 0x00007fb7119537d3 in
clang::CodeGen::CodeGenFunction::EmitCompoundStmtWithoutScope
(this=0x7fffd4fdc7e0, S=...,
    GetLast=false, AggSlot=...) at
/u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CGStmt.cpp:453
#38 0x00007fb7119d69ec in
clang::CodeGen::CodeGenFunction::EmitFunctionBody
(this=0x7fffd4fdc7e0, Body=0x4a695e0)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenFunction.cpp:1147
#39 0x00007fb7119d75bc in
clang::CodeGen::CodeGenFunction::GenerateCode (this=0x7fffd4fdc7e0,
GD=..., Fn=0x4699718, FnInfo=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenFunction.cpp:1312
#40 0x00007fb7119facdf in
clang::CodeGen::CodeGenModule::EmitGlobalFunctionDefinition
(this=0x4b103a0, GD=..., GV=0x4699718)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenModule.cpp:4528
#41 0x00007fb7119f3f73 in
clang::CodeGen::CodeGenModule::EmitGlobalDefinition (this=0x4b103a0,
GD=..., GV=0x4699718)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenModule.cpp:2890
#42 0x00007fb7119f0ec2 in clang::CodeGen::CodeGenModule::EmitDeferred
(this=0x4b103a0)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenModule.cpp:2208
#43 0x00007fb7119e8448 in clang::CodeGen::CodeGenModule::Release
(this=0x4b103a0)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenModule.cpp:399
---Type <return> to continue, or q <return> to quit---
#44 0x00007fb71166b523 in (anonymous
namespace)::CodeGeneratorImpl::HandleTranslationUnit (this=0x4ceb9c0,
Ctx=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/ModuleBuilder.cpp:267
#45 0x00007fb71163ff48 in
clang::BackendConsumer::HandleTranslationUnit (this=0x49fc8d0, C=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:283
#46 0x00007fb711dbf5f5 in
clang::MultiplexConsumer::HandleTranslationUnit (this=0x2cc6290,
Ctx=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/Frontend/MultiplexConsumer.cpp:292
#47 0x00007fb71203fb15 in clang::ParseAST (S=..., PrintStats=false,
SkipFunctionBodies=false)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/Parse/ParseAST.cpp:171
#48 0x00007fb711d7fee7 in clang::ASTFrontendAction::ExecuteAction
(this=0x41bcf20)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1058
#49 0x00007fb71163b730 in clang::CodeGenAction::ExecuteAction (this=0x41bcf20)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/CodeGen/CodeGenAction.cpp:1184
#50 0x00007fb711d80137 in clang::WrapperFrontendAction::ExecuteAction
(this=0x4cb8cd0)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/Frontend/FrontendAction.cpp:1089
#51 0x00007fb711d7f83f in clang::FrontendAction::Execute (this=0x4cb8cd0)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/Frontend/FrontendAction.cpp:950
#52 0x00007fb711d0d96f in clang::CompilerInstance::ExecuteAction
(this=0x7fffd4fe1ed0, Act=...)
    at /u/geoff/llvmrepo/llvm-project/clang/lib/Frontend/CompilerInstance.cpp:984

Hi Geoff,

Nothing in that backtrace leaps out at me. Based on the stack trace and description my first guess would be a clang misconfiguration rather than a JIT bug.

How is that clang invocation being made? Is it from inside a callback from ORC, or is it before you add your module to the JIT?

– Lang.

Thanks, Lang.

I think I have identified the commit that breaks our application,
although I haven't quite finished verifying (it takes a while to
compile and test a debug version of LLVM). It's commit
1a4fb2edcb908d6c9141036d29b46a347b1b6f18, dated June 1:

Hi,

That patch was from an ongoing effort to consolidate OpenMP generation in clang. If memory serves the implementation there is still a little incomplete. It’s supposed to use types from OMPConstants rather than ones it defined itself and the methods used to create the functions shouldn’t need to be static. However attempting this caused a lot of errors so there might be an underlying problem related to the issues you’re seeing. Another one of the errors I had to deal with was dealing with conflicting function declarations, for example declaring omp_get_thread_num extern in the file. Judging from the assertions you got previously this might be the problem. What would happen is if the name of an OpenMP runtime function was specified in the file the types wouldn’t match up when it tried to find the declaration and cause an error. The solution to this was to always attempt to typecast the function type found in the module to the one defined in OMPKinds.def. There should be some debug clauses there that will tell you the types it’s finding if that’s the issue.

Joseph Huber

This looks strange but it makes sense that you see the assertion "because of" that patch.

We use this code path to generate OpenMP code for a while nowthat is why I am not convinced there is not another (undiagnosed) problem which is the actual cause.

Could you run it with `-debug-only='openmp-opt,openmp-ir-builder'` please? Is it possible you merged in an old module or some user code with OpenMP declarations?

Thanks,

Johannes

Thanks, Joseph and Johannes.

I have not merged in anything, I am using the code from the repository
as is. What is this -debug-only option, and to whom would I pass it? I
am running our own JIT application, which uses clang to compile
modules on the fly via clang::CompilerInstance::ExecuteAction().

Working on the assumption that there is a mismatch in the declared
type of an OpenMP runtime function, can you give me a hint as to where
I might find which function it is, and where it would be declared?
What I am seeing in gdb is that a two-parameter function is indeed
being passed two arguments, but the first argument is not of the
expected type. The call originates in
CGOpenMPRuntime::emitForStaticFinish(), with DKind ==
llvm::omp::Directive::OMPD_for, at this line:

  CGF.EmitRuntimeCall(OMPBuilder.getOrCreateRuntimeFunction(
                          CGM.getModule(), OMPRTL___kmpc_for_static_fini),
                      Args);

But I don't know where to go from there... Help?

Geoff

I am looking into a "fix", though I'm not 100% sure what this problem actually is.

Can you dump the two first arguments, I mean the llvm::Values and the llvm::Types? I would

"expect" they are equal structs with different names. If that is the case, my "fix" should

make this work. If not, I'm a little lost. We can always go the route of Clang, simply type

cast the function pointer, that is not great either though.

Yep, it happens three times, then crashes afterwards, since I removed
the assert...

arg 0: expected %struct.ident_t*
got %struct.ident_t.21*
value @0 = private unnamed_addr global %struct.ident_t.21 { i32 0, i32
514, i32 0, i32 0, i8* getelementptr inbounds ([23 x i8], [23 x i8]*
@.str, i32 0, i32 0) }, align 8

arg 0: expected %struct.ident_t*
got %struct.ident_t.21*
value @1 = private unnamed_addr global %struct.ident_t.21 { i32 0, i32
66, i32 0, i32 0, i8* getelementptr inbounds ([23 x i8], [23 x i8]*
@.str, i32 0, i32 0) }, align 8

arg 0: expected %struct.ident_t*
got %struct.ident_t.21*
value @2 = private unnamed_addr global %struct.ident_t.21 { i32 0, i32
2, i32 0, i32 0, i8* getelementptr inbounds ([23 x i8], [23 x i8]*
@.str, i32 0, i32 0) }, align 8

Call parameter type does not match function signature!
%struct.ident_t.21* @2
%struct.ident_t* call void (%struct.ident_t*, i32, void (i32*, i32*,
...)*, ...) @__kmpc_fork_call(%struct.ident_t.21* @2, i32 4, void
(i32*, i32*, ...)* bitcast (void (i32*, i32*, i64*,
%"class.std::vector.22"*, %"class.std::vector.22"*, float*)*
@.omp_outlined. to void (i32*, i32*, ...)*), i64* %_n,
%"class.std::vector.22"* %5, %"class.std::vector.22"* %6, float*
%b.addr)
in function _ZN39kp_8d75c6d9_5169_4aa9_9900_7e97ce4e9b897executeERKSt6vectorIfSaIfEEfRS2_PN5kirpi11GraphTimingE
LLVM ERROR: Broken function found, compilation aborted!
Abort (core dumped)

Ok, `struct.ident_t.21` looks like a "duplicate" of `struct.ident_t`, which is the problem here.

I'm testing my fix right now. Will let you know in a bit.

Yeah, I remember encountering that error before when getting it to pass the libomp test suite. If you have a struct named “ident_t” somewhere the compiler will rename it because of the conflict with the runtime declaration. This should be solved by casting the usage to the function type found in the definition (i.e. bitcasting a struct.ident_t.21 to struct.ident_t) which solved the issues with the libomp test cases. I’m not sure why it’s showing up in this case however.

This patch should address the issue:

https://reviews.llvm.org/D80735

With it, we should not create a second `struct.ident_t` in clang.

I hop to get it approved today or tomorrow.

The patch does seem to work for us, thank you! I do hope you can get
it into the 11.x branch...

Geoff

I'm not sure if we need to file a bug but we might want to go that route, especially since Hans in on vacation.

So could you file a bug, brief description, we cc Hans and ask the fix to be included in the release. WDYT?

~ Johannes