Out-of-line atomics implementation ways

Greetings everyone,
I am working on Aarch64 LSE out-of-line atomics support in LLVM, porting this GCC series: Richard Henderson - [PATCH, AArch64 v4 0/6] LSE atomics out-of-line
After local design experiments I've got some questions about upstream-suitable ways of implementation. More specifically:

1. Pass to expand atomics to library helper functions calls.
These helpers test for the presence of LSE instructions and dispatch to corresponding sequence of instructions.
There are 100 helpers resulting from various combinations of instruction = { cas| swp | ldadd | ldset| ldclr| ldeor }, memory model = { relax, acq, rel, acq_rel } and size = {1, 2, 4 , 8, 16}.
I am considering two possibilities:
i. Atomic Expand pass: add new AtomicExpansionKind::OutOfLine, and if it was set by target expand atomics to RTLIB libcalls. It will require to add 100 new "standardized" library names to RuntimeLibcalls.def and redefine them for Aarch64 target to comply with libgcc implementation ( like this : "cas4_relax" -> " __aarch64_cas4_relax" )
ii. Lower atomics in question later on Instruction Selection pass: for Aarch64 out-of-line atomics targets replace atomicrmw/cmpxchg to __aarch64 helpers libcalls. Then there is no need for runtime library calls extension, however this approach potentially gives less opportunity for compiler optimizations and appears to be more aarch64 specific.

2. Way of generating helpers code
To generate mentioned helpers and calls, preprocessor macros were used on gcc side and 'foreach' make targets on libgcc part of interface.
Concerning LLVM, compiler-rt library builtins readme states: "Each function is contained in its own file. Each function has a corresponding unit test under test/Unit." and compilation is controlled by cmake. In addition, RuntimeLibcalls.def contains HANDLE_LIBCALL macros and would need to be redesigned to include compile time names generation.
So I have a choice of:
i. Slightly change/review some LLVM concepts and generate helpers on the fly.
ii. Prepare all code locally and commit 100+ lines of names and files.

I would greatly appreciate any thoughts, suggestions and comments to create easy-to-go upstream patch,
Thanks in advance,
Pavel

Greetings everyone,
I am working on Aarch64 LSE out-of-line atomics support in LLVM, porting this GCC series: https://gcc.gnu.org/legacy-ml/gcc-patches/2019-09/msg01034.html
After local design experiments I’ve got some questions about upstream-suitable ways of implementation. More specifically:

Great!

  1. Pass to expand atomics to library helper functions calls.
    These helpers test for the presence of LSE instructions and dispatch to corresponding sequence of instructions.
    There are 100 helpers resulting from various combinations of instruction = { cas| swp | ldadd | ldset| ldclr| ldeor }, memory model = { relax, acq, rel, acq_rel } and size = {1, 2, 4 , 8, 16}.
    I am considering two possibilities:
    i. Atomic Expand pass: add new AtomicExpansionKind::OutOfLine, and if it was set by target expand atomics to RTLIB libcalls. It will require to add 100 new “standardized” library names to RuntimeLibcalls.def and redefine them for Aarch64 target to comply with libgcc implementation ( like this : “cas4_relax” → " __aarch64_cas4_relax" )
    ii. Lower atomics in question later on Instruction Selection pass: for Aarch64 out-of-line atomics targets replace atomicrmw/cmpxchg to __aarch64 helpers libcalls. Then there is no need for runtime library calls extension, however this approach potentially gives less opportunity for compiler optimizations and appears to be more aarch64 specific.

Acting like you have LSE instructions for AtomicExpandPass’s shouldExpand* functions, so that it doesn’t do any IR expansion, and then handling the resulting ATOMIC_LOAD_*/etc ISD nodes in isel is definitely the better choice. Lowering to a runtime lib call in ISEL is simple and straightforward, and exactly what you need.

  1. Way of generating helpers code
    To generate mentioned helpers and calls, preprocessor macros were used on gcc side and ‘foreach’ make targets on libgcc part of interface.

Concerning LLVM, compiler-rt library builtins readme states: “Each function is contained in its own file. Each function has a corresponding unit test under test/Unit.” and compilation is controlled by cmake.

You’ll definitely want to have each function end up in its own object-file. Probably the simplest way to accomplish that is to have separate source files for each function, each one containing simply some #defines to select the variant, then #include “lse_impl.inc” – or something along those lines. Making the buildsystem responsible for that is likely also possible…but meh…not sure it’s worth it.

In addition, RuntimeLibcalls.def contains HANDLE_LIBCALL macros and would need to be redesigned to include compile time names generation.

No redesign would be needed here. This file is preprocessed; you can #define an internal helper macro which calls HANDLE_LIBCALL multiple times, if desirable. However, it’s probably also fine to just list them all out.

Current precent in the codebase is the __sync_* libcalls. They have essentially the semantics you want, except that they're all seq_cst.

On the LLVM side, I'd rather not have two ways to do the same thing, so I'd prefer to extend the existing mechanism. Adding 100 lines to RuntimeLibcalls.def seems a bit unfortunate, but I think you can reduce that using some C macros.

On the compiler-rt side, given the large number of functions, some CMake magic might be appropriate.

-Eli

Thank you for useful advice,

Outline atomics are implemented in two patches:

https://reviews.llvm.org/D91156 - compiler-rt part

https://reviews.llvm.org/D91157 - llvm+clang part

I’ve tested it on aarch64+lse linux board (libc++ and libstdc++) and

checked compilation of Chromium and AOSP hacked build with updated libgcc supporting outline atomics helpers

(to do fully Android testing libgcc should be updated to version 10 or replaced by compiler-rt with my patch above)

I would be very grateful for review and comments,

Pavel