I'm working on using LLVM as a back-end for the Haskell GHC compiler. As part of that work I have modified LLVM to include a new custom calling convention for performance reasons as outlined previously in a conversation on this mailing list:
This custom calling convention on x86-32 needs to handle just 4 parameters, passing them in %ebx, %ebp, %esi, %edi. These are all callee saved registers. To implement the custom calling convention I change llvm in two places:
1) X86CallingConv.td : Add new calling convention.
2) X86RegisterInfo.cpp : Modify 'getCalleeSavedRegs' to remove the above registers from the callee saved registers.
This works fine mostly. On Linux, the generated code passes the GHC testsuite. On Mac however the GHC testsuite fails on any code which uses the ffi (which is implemented by libffi [http://sourceware.org/libffi/\]). Programs which fail segfault with the error '__dyld_misaligned_stack_error'. The issue seems to be from my investigations that the ffi call should be 16-byte aligned as per Mac OS X's ABI.
I'm hoping someone is able to confirm that my changes would have introduced this bug and how to go about fixing it.
Another minor issue is that the generated code has a strong tendency to manipulate the stack pointer when its not required. For a large amount of functions, the generated code will start and finish with sp manipulation to give the function some space despite the function not otherwise using the stack.
subl $4, %esp
movl 4(%ebp), %eax
movl 8(%ebp), %ecx
movl (%ebp), %esi
movl %eax, 8(%ebp)
movl %ecx, 4(%ebp)
addl $4, %ebp
addl $4, %esp
It would be nice to fix this up as well.