I'm sending this discussion to both wine-devel and llvm-dev, to try to keep the discussion open for both sides, to try to find a workable compromise. This was preliminarily discussed on llvm-dev already a few weeks ago.
One of the major unresolved issues with Wine for AArch64 is how to handle the platform specific register x18. (https://bugs.winehq.org/show_bug.cgi?id=38780) As Windows on AArch64 is commercially available for over a year now, it'd be nice to get this issue settled by at least some sort of compromise.
On AArch64, x18 is a platform specific register; some platforms treat it as reserved for platform specific use, while others (Linux) don't and treat it as any free temporary register. On Windows, it is used to hold the TEB pointer.
When calling Wine builtin functions from the Windows native code, the Wine builtin functions can clobber the x18 register, as the Wine code is built for the ABI and calling conventions used on Linux. This part is easy to work around by compiling Wine with the flag -ffixed-x18, which makes the compiler treat this register as reserved, avoiding clobbering it.
However, as the Wine code may end up calling functions in the surrounding host environment (mainly the libc, but potentially also other feature libraries that are used), and these libraries have not been built with the -ffixed-x18 flag.
There are a few different options for going about this matter:
1) Rebuild the surrounding host environment (the whole linux distribution) with -ffixed-x18. This would be the perfect solution, but is highly impractical for general use of Wine by regular users in their existing setups. (Convincing major distributions to start configuring their compilers in this way also doesn't seem to be happening.)
2) Force relays on entry to Wine builtins, where the relay can restore x18 to the right TEB pointer on return. This is pretty simple and straightforward, but not very elegant. One benefit is that this doesn't need to back up the original value of x18 but can always refetch the right TEB pointer value and set it. A patch to this effect have been sent before (https://source.winehq.org/patches/data/137759) but wasn't picked up.
One major limitation is that while this ensures the right value in x18 on return from the Wine functions, it doesn't help restoring x18 to the right value before calling callback functions.
I've run with this approach for nearly two years, without running into any issues with it (with the limited amount of code I run in wine; I don't think any of the command line executables I run in wine use callbacks though).
3) Enhance the compiler to automatically back up x18 on entry to functions marked with __attribute__((ms_abi)) and restore it on return. This would have the same effect as 2) above (but more elegantly), but also with the same limitations wrt callbacks.
(Using this with Wine requires changing where x18 is initialized, because when starting a process, x18 can get clobbered after signal_init_thread is called, before handing control over to the native code. A patch for this has been sent at https://source.winehq.org/patches/data/164651.)
A proof of concept/RFC patch for LLVM to implement this feature has been sent at https://reviews.llvm.org/D61892.
4) Enhance the compiler to back up x18 in every function (if necessary) to restore it after every function call. (Except function calls to functions in the same translation unit, which can be expected to not clobber the register.) This should achieve almost everything; x18 is maintained containing the right value throughout the Wine code as far as possible, like when doing callbacks.
If callbacks to native code is made from within a callback from an external library (e.g. qsort in libc), x18 can be lost though. I guess this isn't a common pattern for libc functions at least, but I have no idea if it's more common for other external feature libraries that Wine uses.
A proof of concept/RFC patch for LLVM to implement this feature has been sent at https://reviews.llvm.org/D61894. Do note that this approach can be pretty controversial to upstream to LLVM.
This also requires the same Wine patch as 3), but for a different reason. When signal_init_thread is called to initialize x18, the caller will restore the register on return. The most robust way to intitialize it is right before handing control over to native code.
5) Enclose every callback call in Wine with a wrapper/thunk that sets up the register correctly. This would be a perfect solution, but is practically unfeasible. As far as I know, this is the approach that was used for Win16 back in the day, calling WOWCallback16Ex every time Wine code should call back into Win16 code. Given the size of Wine today and the number of different places where callbacks are made (where the function pointers are called without any extra wrapping), this is unfeasible (and I have a very hard time seeing such a patch accepted into Wine).
What are the potential acceptable compromises for taking this matter forward?