Is LLVM Flang runtime thread-safe?

The following test case is captured in WRF 4.2.

$ cat m.f90 
Module test
private
public ESMF_TimeGet

type, public :: e_time
  integer :: year, month, dayofmonth, hour, minute, second
end type

contains

subroutine ESMF_TimeGet(time, timeString)
   type(e_time) :: time
   character (len=*), intent(out), optional :: timeString
   integer :: year, month, dayofmonth, hour, minute, second
   year = time%year
   month = time%month
   dayofmonth = time%dayofmonth
   hour = time%hour
   minute = time%minute
   second = time%second
   if (present(timeString)) then
     CALL ESMFold_TimeGetString( year, month, dayofmonth, &
                                 hour, minute, second, timeString )
   endif

end subroutine ESMF_TimeGet

subroutine ESMFold_TimeGetString( year, month, dayofmonth, &
                                  hour, minute, second, TimeString)

      integer, intent(in) :: year
      integer, intent(in) :: month
      integer, intent(in) :: dayofmonth
      integer, intent(in) :: hour
      integer, intent(in) :: minute
      integer, intent(in) :: second
      character*(*), intent(out) :: TimeString
      write(TimeString,FMT="(I4.4,'-',I2.2,'-',I2.2,'_',I2.2,':',I2.2,':',I2.2)") &
             year,month,dayofmonth,hour,minute,second

end subroutine ESMFold_TimeGetString
end
$ cat test.f90 
program main
  use test
  call foo(10)
contains
  subroutine foo(n)
    type(e_time) :: t = e_time(2022, 10, 30, 03, 30, 20)
    character(100) :: s
    integer :: n
    !$omp parallel
    !$omp do
    do i = 1, n
      call ESMF_TimeGet(t, s)
      print *, s
    enddo
    !$omp end do
    !$omp end parallel
  end
end

The compilation method in WRF settings is as follows:

$ export OMP_NUM_THREADS=10
$ gfortran m.f90 -c
$ gfortran test.f90 -c -fopenmp
$ gfortran test.o m.o -fopenmp
$ ./a.out 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20                                                                                 
 2022-10-30_03:30:20      
$ ifort m.f90 -c
$ ifort test.f90 -c -fopenmp
$ ifort m.o test.o -fopenmp
$ ./a.out 
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20                                                            
                      
 2022-10-30_03:30:20     
$ flang-new m.f90 -c
$ flang-new -fopenmp test.f90 -c
$ flang-new -flang-experimental-exec -fopenmp test.o m.o 
$ export OMP_NUM_THREADS=1
$ ./a.out 
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
 2022-10-30_03:30:20                                                           
                       
$ export OMP_NUM_THREADS=10
$ ./a.out 

fatal Fortran runtime error
fatal Fortran runtime error 2022-10-30_03:30:20                                                           
                       

fatal Fortran runtime error
fatal Fortran runtime error
fatal Fortran runtime error(./test.f90:13): 
fatal Fortran runtime error
fatal Fortran runtime error(./test.f90:13): Could not acquire exclusive lock on unit 6, perhaps due to an attempt to perform recursive I/O
Could not acquire exclusive lock on unit 6, perhaps due to an attempt to perform recursive I/O
(./test.f90:13)Aborted (core dumped)   

As @kiranchandramohan mentioned, some compilers support thread-safe IO runtime such as gfortran, xlf.

Currently, LLVM Flang crash in IO runtime. So, should LLVM Flang also support thread-safe IO runtime?

The runtime is using pthread_mutex. Are Fortran OpenMP threads mapped to distinct pthreads?

Is the application attempting to perform recursive I/O? Perhaps a call stack traceback can shed some light on the root cause of the error message.

I am not an expert in OpenMP runtime. It seems that OpenMP runtime in LLVM does not use pthread_mutex_t for threads synchronization. llvm-project/kmp_lock.h at main · llvm/llvm-project · GitHub

Is the application attempting to perform recursive I/O? Perhaps a call stack traceback can shed some light on the root cause of the error message.

No. I update the test results for LLVM Flang. When there is only 1 thread, it works OK.

Something simple like the following is failing currently. I think this used to work before.

program main
  integer :: s
  !$omp parallel
    print *, s
  !$omp end parallel
end

It seems this started failing after the following commit. Have not investigated whether we should make the check more precise before issuing the recursive I/O error.

commit 921316af6e6b34d57d5e8f26f86f2bce8d034c7f (HEAD)

    [flang][runtime] Catch & report attempts at recursive I/O
    
    When an I/O statement contains a function call that attempts
    to perform I/O on the same unit, detect the recursive I/O
    and terminate with a useful message rather than deadlocking in
    the threading library.
    
    Differential Revision: https://reviews.llvm.org/D131097
1 Like

There is a TODO in this patch, implementing this (I think it is checking per thread) possibly will fix the issue.

  // TODO: replace with a thread ID
  bool isBusy_{false}; // under lock_
1 Like
  template <typename A, typename... X>
  IoStatementState &BeginIoStatement(const Terminator &terminator, X &&...xs) {
    lock_.Take(); // release in EndIoStatement()
    A &state{u_.emplace<A>(std::forward<X>(xs)...)};
    if constexpr (!std::is_same_v<A, OpenStatementState>) {
      state.mutableModes() = ConnectionState::modes;
    }
    directAccessRecWasSet_ = false;
    io_.emplace(state);
    return *io_;
  }
...
void ExternalFileUnit::EndIoStatement() {
  io_.reset();
  u_.emplace<std::monostate>();
  lock_.Drop();
}

This error can be fixed by making IoStatement single threaded, but i’m not entirely sure if this is the intended way to fix it.

Fix

1 Like