Atomics on Windows

Subprojects Flang
1

Hello,
I built the LLVM master branch with Flang on WoA last week and tried to compile the MUMPS software. The code contains clauses like these:

!$OMP ATOMIC CAPTURE
!$OMP ATOMIC WRITE
!$OMP ATOMIC READ
!$OMP ATOMIC UPDATE
!$OMP END ATOMIC

The source code can be obtained from here: http://deb.debian.org/debian/pool/main/m/mumps/mumps_5.7.3.orig.tar.gz

The code compiles, but I am encountering a link-time error:

error LNK2019: unresolved external symbol __atomic_load referenced in function _QFzmumps_arrow_try_treat_recv_bufPzmumps_arrow_treat_recv_buf
error LNK2019: unresolved external symbol __atomic_compare_exchange referenced in function _QFzmumps_arrow_try_treat_recv_bufPzmumps_arrow_treat_recv_buf

Which runtime library provides __atomic_load and __atomic_compare_exchange on Windows? Isn’t this supposed to be handled automatically by the compiler, as it is with other compilers (like ifort)?

2

These symbols should come from compiler-rt. Are you building compiler-rt?

3

Yes, I build LLVM with -DLLVM_ENABLE_PROJECTS="clang;flang;openmp" -DLLVM_ENABLE_RUNTIMES="compiler-rt"

In my installation directory, under lib/clang/20/lib/windows, I have the following files:

clang_rt.asan_dynamic_runtime_thunk-aarch64.lib
clang_rt.asan_dynamic-aarch64.dll
clang_rt.asan_dynamic-aarch64.lib
clang_rt.asan_static_runtime_thunk-aarch64.lib
clang_rt.builtins-aarch64.lib
clang_rt.profile-aarch64.lib
clang_rt.stats_client-aarch64.lib
clang_rt.stats-aarch64.lib
clang_rt.ubsan_standalone_cxx-aarch64.lib
clang_rt.ubsan_standalone-aarch64.lib
4

I think you need a few flags to switch on the atomic libraries in compiler-rt.

    -DCOMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF
    -DCOMPILER_RT_BUILD_STANDALONE_LIBATOMIC=OFF
    -DCOMPILER_RT_USE_ATOMIC_LIBRARY=ON

You might also need the following.

    -DCOMPILER_RT_BUILD_BUILTINS=ON
    -DCOMPILER_RT_USE_BUILTINS_LIBRARY=ON
5

Thanks! This solved the problem. But I came across a different issue at runtime:

fatal Fortran runtime error(C:\.conan\b2694b\1\llvm-project\flang\runtime\descriptor.cpp:71): not yet implemented: type category(6)

This didn’t occur in LLVM 19.

6

If you have a small reproducer then please file a github issue.

7

In theory, on Windows LLVM should only use libintrinsic calls from msvcrt/ucrt, but if there is no equivalent still emits symbols using gcc’s ABI (implemented in libgcc/libatomic or clang_rt.builtins).

This should be addressed with https://github.com/llvm/llvm-project/pull/110217 which adds an implicit dependency to clang_rt.builtins. The dependency should have already been added implicitly into flang-generated .obj files.

So __atomic_load and __atomic_compare_exchange should be present in clang_rt.builtins-<arch>.lib. You can check with DUMPBIN /SYMBOLS. Msvc comes with its own version of it if you don’t compile it youself, but the Flang driver unfortunately does not find it by default ([flang][compiler-rt] incompatible names for clang_rt.builtins · Issue #95698 · llvm/llvm-project · GitHub).

I don’t know about the not yet implemented: type category(6) error.

8

I checked dumbin myself and __atomic_load wasn’t part of flang_rt.builtins by default (only __atomic_load_n).

It is either added with COMPILER_RT_EXCLUDE_ATOMIC_BUILTIN=OFF (as @kiranchandramohan suggested), or provided by clang_rt.atomic (enabled with COMPILER_RT_BUILD_STANDALONE_LIBATOMIC=ON and needs to be added manually to the flang command line).

So by default it is not present anywhere. For Clang the intention seems to be that the libcall should not be needed on Windows, see pragma pack struct + volatile results in an unresolved external "atomic_load" when using the Microsoft Linker · Issue #28217 · llvm/llvm-project · GitHub. Currently, Flang falls back to libcall intrinsics more often than necessary, the PR Add emitAtomicCompareExchangeBuiltin helper function by Meinersbur · Pull Request #101966 · llvm/llvm-project · GitHub might solve this by not falling back to libcalls when not necessary.

9

I think that there might be something more going on here. CAS is definitely something that MSVC can emit. We should see what MSVC is generating under similar conditions in C. I don’t expect to see a call to __atomic_compare_exchange as there is at the very least _InterlockedCompareAndExchange.

__atomic_load I can potentially see being formed, though, Microsoft has added support for C11 atomics and so ideally we should be treating those as _Atomic instead.

10

MSVC only natively supports 1/2/4/8 atomics. MSVC std::atomic uses locks to implement sizes other than 1/2/4/8, and _Atomic doesn’t work at all with other sizes. (See also C11 Atomics in Visual Studio 2022 version 17.5 Preview 2 - C++ Team Blog )

1 Like
11

I updated the PR that should fix the the use of non-existant libcall atomics: [Transforms] Introduce BuildBuiltins.h atomic helpers by Meinersbur · Pull Request #101966 · llvm/llvm-project · GitHub

That is, this should always use the LLVM instruction instead the libatomic (which mat not exist on Windows) for standard sizes 1, 2, 4, 8 bytes, including Windows. For 16 bytes (e.g. complex double), I think it depends on hardware support of atomics. For all other sizes, Windows, we might need to inline-emit code that uses a lock, like @efriedma-quic mentioned happened in Microsofts implementations of std::atomic. The linked blog entry also mentions “support rountines”, maybe we could call those. With some experiments on godbolt one can see that msvc generates calls to functions such as _Atomic_lock_and_store declared here: windows-msvc-sysroot/include/vcruntime_c11_atomic_support.h at main · trcrsired/windows-msvc-sysroot · GitHub (i.e. Microsoft’s libatomic)