How to add a new calling convention?

Hello Clang-Devs,

this question originally came from the LLVM Discord server:

Currently we are using an inhouse library which a different department developed for us. I’m not aware about what they are doing in there implementation, however whenever we use functions from said library we are instructed to save the GS register. Currently we manage this by using inline assembly, we push the register before the call, do the call and pop it back. However this is very error prone because you can easily forget to do so.

My hope was to teach Clang an additional calling convention which we could explicitly declare with “__specialcall” or something. That convention would push the GS register for us and then do the normal x86_64 calling convention. In Discord I was told that this is possible but that there are no instructions for this.

Can someone guide me in the right direction for doing such a thing? Hopefully in a way a beginner can pick it up >o<

Thank you in advance and kind greetings

Björn

Hello Clang-Devs,

this question originally came from the LLVM Discord server:

Currently we are using an inhouse library which a different department developed for us. I'm not aware about what they are doing in there implementation, however whenever we use functions from said library we are instructed to save the GS register. Currently we manage this by using inline assembly, we push the register before the call, do the call and pop it back. However this is very error prone because you can easily forget to do so.

Ew, that's horrible. (I recall having to do it once when a library clobbered SI & DI, which are supposed to be callee saved.)

My hope was to teach Clang an additional calling convention which we could explicitly declare with "__specialcall" or something. That convention would push the GS register for us and then do the normal x86_64 calling convention. In Discord I was told that this is possible but that there are no instructions for this.

Can someone guide me in the right direction for doing such a thing? Hopefully in a way a beginner can pick it up >o<

You're going to have to modify the X86 code generation directly. Mark the functions of interest with some kind of new attribute, and then look for that in the call generation code.

llvm/lib/Target/X86/X86ISelLowering.cpp is the file of interest there, make GS conditionally a caller-saved register in some way. More than that I don't know.

(Just in case you're unaware, GS is used to access TLS data in the x86 ABI. Not sure if that's true on windows though.)

Adding calling conventions is actually quite awful, you can find my old patches implementing RegCall on Phab somewhere if you'd like some examples.

A big part of the problem is that we store calling conventions directly as a part of the FunctionType, so every one you need to add increases the size of FunctionType (and therefore harms the whole compiler, even if not used), which is quite unfortunate to have to do.[0]. You DO need it to be in the type system (even in your example) because pointers to these functions need to correctly copy/have the correct semantics.

Then, once you get it into the type system correctly, you need to handle it in clang-codegen, in the lib/CodeGen/TargetInfo.cpp versions of EACH target that you want to support it on. This code itself is actually quite terrible in itself, as it is quite disorganized and unmaintained. Additionally, X86/X86-64 is actually implemented across FOUR different types there, since there are those plus bifurcated for Linux and Windows. This is less of a concern for YOUR type, since the new 'calling convention' is actually not affecting the normal calling mechanism (that is, it doesn't change how parameter types are passed), but you'll still want to make sure we label the function types correctly in the IR.

THEN, you'd need to get it to be in the IR, and do what Nathan says below. I'd STRONGLY discourage adding a new calling convention if at all possible...

[0] I have on my backlog to design/implement a mechanism to get some of these bits 'back' for other platforms by making the bits target-specific (so we're not using bits in x86 mode to represent arm calling conventions that are impossible to get, and also to free up the use of these bits for calling conventions of other targets), but I haven't had a chance to get to it yet.

Thank you for the reply guys :smiley:

Seems like adding the calling convention might bring more pain then I thought...
Is it a better idea to tackle this issue on IR level then? Making Clang to not compile to object files, but IR instead, patching the effected calls and then passing it to static compiler? I haven't done such a thing so I honestly have no idea how easy it is to locate the calls that need patching (not every function call would need that) and then actually applying the patch.

Thank you for the help so far!

Kind greetings
Björn

One of the things we've(not sure if the open source repo does it anywhere?) done in the past for situations like this (at least, ones where we know the address of functions won't be taken, either by rule or by calling it Undefined Behavior) is to have the set of functions that this is important for recognized by name in the code generator and have the special handling happen there. This only works because there is a fairly small/finite list of functions.

I will try to solve our GS problem now on a C/C++ level with a wrapper around those functions...
This seems to be easier for now... However... that entire subject is really interesting to me, just over my LLVM abilities...

Thank you everyone! :3

Hi,

Thank you for the reply guys :smiley:

Seems like adding the calling convention might bring more pain then I thought...
Is it a better idea to tackle this issue on IR level then? Making Clang to not compile to object files, but IR instead, patching the effected calls and then passing it to static compiler? I haven't done such a thing so I honestly have no idea how easy it is to locate the calls that need patching (not every function call would need that) and then actually applying the patch.

I think you need to do both parts anyway: clang doesn't generate object code, clang generates LLVM IR and LLVM generates object code. If you want a new calling convention exposed to C/C++ then you first need to expose a new clang calling convention, make clang lower it to a new IR calling convention, and then modify the back ends to handle the new calling convention.

The good news is that if you add the calling conventions to the two enums (for C in clang, for IR in LLVM) and run a build with something like `ninja -k100` then it will tell you all of the places outside of the back end where you need to add handling for your new calling convention. You can probably find a similar example near all of these to copy. I've done this a bunch of times and never felt the need to note down all of the places because the incomplete enum coverage in switch warning finds them all for me.

You need to modify a *lot* of places, but each individual change is typically tiny.

David