Intrinsic opt failure


Working on private backend, based on llvm-3.7

Inside lib/Transforms/InstCombine/InstCombineCompares.cpp there is
attempt to optimize fabs:

        if (F->getIntrinsicID() == Intrinsic::fabs ||
          switch (I.getPredicate()) {
          case FCmpInst::FCMP_OGT:
            return new FCmpInst(FCmpInst::FCMP_ONE, CI->getArgOperand(0), RHSC);


CI->getArgOperand(0) returns operand with type (double*) rather then
double, because in our ABI doubles are passed via memory.

Next assertion fires inside FCmpInst ctor:

llvm::FCmpInst::FCmpInst(llvm::CmpInst::Predicate, llvm::Value*,
llvm::Value*, const llvm::Twine&): Assertion `getOperand(0)->getType()
== getOperand(1)->getType() && "Both operands to FCmp instruction are
not of the same type!"' failed.

I think, here required some check that CI->getArgOperand returns
correct value before trying to optimize.

What would you advise?

The LLVM intrinsics in the LLVM IR must conform to whatever the IR's requirements are. That means that even in the case of your backend, the arguments to the intrinsic must be doubles, not their addresses. You can convert the intrinsics to function calls, together with the call frame set up, etc. during instruction selection.

Check setLibcallName, setLibcallCallingConv (and related functions) in include/llvm/Target/TargetLowering.h. The Libcall enum is defined in include/llvm/CodeGen/RuntimeLibcalls.h



Thanks a lot for correction, I rewritten and debugged some code and
now everything works: I really should process calling convention in
argument lowering inside llvm backend, not in classifyArgumentType
inside ABIInfo inherited class in clang codegen. This is not at all
obvious: correcting arguments at early stages looks simpler, but leads
to tricky problems with high-level IR optimizations.