RBP register is not preserved before a call to function with custom CC


In our project we use our own custom calling convention which uses only integer args passed in registers only, no varargs and it’s fully TC-capable. Somewhat the vanilla LLVM has as HiPE CC.
Our CC has no any callee-saved registers, so CSR_NoRegs_RegMask/SaveList provided in RegisterInfo for our custom calling convention.

Given that I expect that all registers are caller-saved. And this is true if there is no frame pointer used in a caller function. The caller function is a function with standard C calling convention.

Recently I introduced a change in the caller function which materialized frame pointer (frameAddressTaken==true). Now the code generated for the caller function uses %rbp register as a frame pointer for register spilling/reloading. And there is an attempt to reload some register after the call to a function with our custom CC. But since our CC doesn’t save %rbp then it becomes destroyed causing a crash.

Is this a bug or I do something wrong for our CC support?
We currently use LLVM 10, I’m going to give a try to the latest release.

Code samples:
A currently generated code fragment for the caller func:

	movabs	rbx, offset myTable
	mov	r9, qword ptr [rdi + 88]
	mov	r8, qword ptr [rdi + 96]
	mov	rcx, rdi
	call	qword ptr [rbx + rax]  # call to a function with custom CC
	mov	r15d, eax
	mov	r14, qword ptr [rbp + 208] # 8-byte Reload, but %rbp has been destroyed by the call above

What would I expect:

	push rbp
	movabs	rbx, offset myTable
	mov	r9, qword ptr [rdi + 88]
	mov	r8, qword ptr [rdi + 96]
	mov	rcx, rdi
	call	qword ptr [rbx + rax]  # a call to a function with custom CC
	pop rbp
	mov	r15d, eax
	mov	r14, qword ptr [rbp + 208] # 8-byte Reload

You have caused the compiler to select a register that it believes is preserved across subroutine

I have my ABI which has the FP register (when used) as the register immediately prior to the SP
register. Registers R16…R31==SP are preserved (callee saved). So, if FP is required, what used
to be in FP is pushed on the stack as a preserved register, and restored when FP is no longer
in use–thus preserving the compiler implied semantic.

I don’t know about your prototype ABI or your actual ABI; but in general, if the compiler currently
thinks a register is preserved across a subroutine call, it is not likely to be in your best interest to
teach the compiler otherwise.

@Mitch_Alsup, could you please elaborate a bit? It’s not really clear for me why the compiler didn’t preserve frame pointer register in my case. And, more important, how to fix it. :slight_smile:
From my point of view it really looks like a bug but I can’t tell it for sure. I did the similar thing for aarch64 resulting the FP materialized and it works fine there.

I could fix it by creating a pass which would just put push/pop around the call (when caller is ccc and callee is mycc), but it doesn’t look as a proper solution, there should be a more elegant way to make the compiler did this for me.

This is the crux of the problem–if you allow the compiler to choose a register in the preserved
(callee saved) register area, then the prologue has to save that/those registers, and epilogue
has to restore that register–or the code will not work.

I suggest looking at the parts of the code that emit prologue and epilogue.