How to inline functions with different calling convention from caller?

IR & Optimizations
#1

Hi

The code is like A–>B–>C, A calls B and B calls C.

A, B, C are functions.

A and B have different calling convention.

The calling convention of B is added by myself.

In function B, B saves and restores some registers in the prolgue and epilogue.

I modify some code in frame lowering to make B have additional callee saved registers.

After inlining, code becomes A → C,
I expect that the function body of A would be

A:
store reg1
store reg2
store reg3
call C
load reg1
load reg2
load reg3

But the result is, A calls C directly without any prologue/epilogue which are origianlly in B.

In my case, the program crashed because reg1, reg2, reg3 are clobbered by C.

How to make inliner be aware of different calling convention between A and B?

Thanks a lot !!

#2

Calling convention is the property of the call, so the CC of A or B shouldn’t have effect on what prologue/epilogue is generated for the call to C.
Inliner shouldn’t really care about CCs, but they are encoded in the MIR/IR: LLVM Language Reference Manual — LLVM 15.0.0git documentation

#3

It sounds like C needs a different calling convention to reflect the fact that it clobbers those registers (maybe instead of B, maybe as well as).