LLVM backend: Treat some function calls specially


I am writing an LLVM backend that generates byte code for a custom virtual
machine. Standard function calls are lowered to a simple
   CALL $offsetInByteCode
This works fine so far using the standard machinery of LLVM. But some
functions are not implemented in byte code, but delegated to native
implementations within the VM. Calls to these functions use a non-standard
calling convention and should be lowered to another opcode:
   CALL_NATIVE $functionIndex
where $functionIndex is a magic byte telling the VM which native function
should be called. As the set of native functions and function indices will
be extended over time, it should not be hardcoded into the backend but
given as input at compile time.

I have some vague ideas on how I could implement this, but would like to
ask for your ideas in order to keep me from going too far in a
wrong/cumbersome direction.

The first option that came into my mind is adding a new intrinsic
"llvm.myVM.call_native(i32 %funcIdx, ...)" and then linking at compile
time a module that looks like
   define void @foo(i32 %a) {
      call @llvm.myVM.call_native(i32 1, i32 %a)
   define void @bar(i32 %a, i32 %b) {
      call @llvm.myVM.call_native(i32 2, i32 %a, i32 %b)
But here I am unsure how to handle the return type of this intrinsic (can
it be overloaded to accept any return type?) and whether I can reuse the
TargetLowering::LowerCall and CCState::AnalyzeFormalArguments stuff.

Maybe another option is to add some kind of metadata to these function
declarations and then check in "LowerCall" whether these metadata flags
are present. If so, then extract the funcIdx from metadata and lower this
call in a special way. If this is doable, are there some example of how I
can do that (or similar things)?

Maybe, you have an even better idea?

Many thanks in advance,

Why not lower everything to a single call during instruction selection, and then write a machine function pass that translates the normal CALL's into CALL_NATIVE?


Hi Micah,

the problem is that calling conventions differ ("i32 and i64 mixed" versus "everything promoted to i64")
and I don't want to manually fix type promotion in a machine function pass, when I get can this for free
in "LowerCall". Indeed, I can lower already some (small hard coded set of) native functions in "LowerCall".

All that's missing is a mechanism to tag some functions as native (and pass the function index) and
access this information during "LowerCall". By the way, this problem of knowing which function is native,
would still be the same in a machine function pass, wouldn't it?