Porting LLVM to OpenVMS

Dear community,

My company is interested in porting LLVM to OpenVMS. To do so, we need to look at extending the AMD64 calling convention to suit the existing calling conventions available on OpenVMS.

We realize LLVM is a big and complex project and for some people who has not worked on it before, it’s going to be a big and tough job that is going to take a lot of time. Still, the job has to be done and we have to start somewhere, so any help or direction would be great.

Now, as to what we need:

  • To pass the argument count to the callee

  • Reserving registers for mapping variables directly to them

We need to look at passing the argument count (i.e. number of arguments passed) to functions in some register. One way that we have considered is to use the second byte of AL to hold the argument count. Do you think that would work well with how LLVM works today? Or is there an alternative way that you would suggest?

We also need to look at reserving registers because existing OpenVMS code relies on being able to map variables directly to registers so that multiple functions can share variables directly inside registers. We may need to reserve up to as many as 6 registers. Do you think this might work? Or will this make codegen bad? We have also considered trying to put these variables into an array and reserving a global variable to index into this array, thereby emulating this feature. Do you think this will work better?

It would be great to get some information to cut down on the area we need to concentrate on to get started. Any help or direction would be great.

image001.png

We need to look at passing the argument count (i.e. number of arguments
passed) to functions in some register. One way that we have considered is
to use the second byte of AL to hold the argument count. Do you think that
would work well with how LLVM works today? Or is there an alternative way
that you would suggest?

Hi Patrik,

Clang/LLVM already supports reserved platform registers, so this would
probably be a number of small changes to both Clang and LLVM to make it
work. But AFAIK, they use entire registers, so you wouldn't have those
registers available to the allocator. This shouldn't be a big problem in
AMD64, though.

We also need to look at reserving registers because existing OpenVMS code

relies on being able to map variables directly to registers so that
multiple functions can share variables directly inside registers. We may
need to reserve up to as many as 6 registers. Do you think this might work?
Or will this make codegen bad? We have also considered trying to put these
variables into an array and reserving a global variable to index into this
array, thereby emulating this feature. Do you think this will work better?

I imagine you'd start by adding the triple in Clang, then walking down the
Driver code to create a toolchain for it (copy what's there for GNU, Apple,
etc). You'll see that different toolchains reserve registers for different
purposes. Re-using the registers other toolchains use would be the easiest,
but that depends if the OpenVMS ABI has any encoded registers already.

Obviously, the more registers you reserve, the more spills you'll have, so
codegen will be affected. But that's a trade-off that you make when
designing the ABI.

I have never designed an ABI myself, but I have been around people that
did, enough to know that it's impossible to get it right, even after 10
years, let alone the first time.

So, if OpenVMS has an ABI, you *MUST* follow it. If it doesn't, than I
strongly encourage you to contact the OpenVMS community *first* and design
an ABI, and only then implement in the compiler.

In LLVM, we usually tend to reject ABI implementations without access to
official documentation for that very reason.

Hope that helps,

cheers,
--renato

I don't see a problem doing any of these (of course it will take some getting used to how all those things work in llvm). You can express storing a number in a fixed register with selection DAG CopyToReg and you can reserve registers.

Look around X86ISelLowering.cpp esp. LowerCall() and X86RegisterInfo::getReservedRegs().

- Matthias

Thank you both for helpful replies J

image001.png