Status of detached tasks in Clang

Dear all,

I am eager to play around with detached tasks in OpenMP. It seems that support has gone into the runtime about a year ago [1] but even with Clang 10 I get a warning that the detach clause is ignored:

$ clang-10 -fopenmp test_detached.c
test_detached.c:25:47: warning: extra tokens at the end of '#pragma omp task' are ignored [-Wextra-tokens]
   #pragma omp task detach(event)

The status website lists detached tasks as work in progress [2]. Is there a way to try detached tasks in some experimental version of Clang? Or am I just missing a flag to enable them in the Clang release?

Thanks a lot,
Joseph

[1] https://reviews.llvm.org/D62485
[2] https://clang.llvm.org/docs/OpenMPSupport.html

Hi Joseph,

I’m not sure who is working on this but we might be able to figure this out on our bi-weekly call Wednesday.

Feel free to call but we should figure this out either way.

Apologies for the delay,

Johannes

Johannes, Alexey,

Thanks for the pointers. I will try to call in to the telco tomorrow.

In the meantime: I was able to install trunk and run tests with detached tasks. However, every once in a while I end up in an assert inside libomp:

OMP: Error #13: Assertion failure at kmp_tasking.cpp(920).

The relevant code is in __kmp_task_finish:

   if (taskdata->td_flags.destructors_thunk) {
     kmp_routine_entry_t destr_thunk = task->data1.destructors;
     KMP_ASSERT(destr_thunk); // <- this assert triggers
     destr_thunk(gtid, task);
   }

My first suspicion is that there is a race condition when one thread fulfills the event while the executing task completes it. I have not been able to spot this issue in the code but maybe someone familiar with it has an idea about what might be going wrong?

Thanks,
Joseph

Alexey,

I am having a hard time trying to come up with a simple reproducer. However, I got hit by an issue that looks similar to what I reported a while ago:
https://bugs.llvm.org/show_bug.cgi?id=37671

I updated it with a simpler reproducer. Maybe this issue causes havoc in my actual application and leads to the assert somehow. Maybe someone can take a look at that #37671 now with the simpler test case?

I will try and work a reproducer for the assert and report back if I find one.

Cheers
Joseph

Ahh, good point! The issue with local variables seems to be specific to untied tasks. I removed all untied tasks in my application but every other run I'm still hitting asserts in the same code path (when calling omp_fulfill_event):

Assertion failure at kmp_tasking.cpp(3744): taskdata->td_flags.complete == 1.
Assertion failure at kmp_tasking.cpp(710): taskdata->td_flags.executing == 0.

Although they are different assertions they are probably just different symptoms of the same problem. My use-case involves MPI (and an experimental extension) so it is hard to boil that down to a simple reproducer.

Just so I know whether it's a direction to investigate: could these asserts be triggered by calling omp_fulfill_event twice on the same event? Or is that case handled explicitly? I don't know how this could happen in my current setup but if it's not handled by libomp I will investigate that possibility.

Thanks for the help so far :slight_smile:

Cheers
Joseph

Hi all,

I looked a bit into this issue. As I understand the code, there is a race between __kmp_task_finish and __kmp_fulfill_event:

https://github.com/llvm/llvm-project/blob/master/openmp/runtime/src/kmp_tasking.cpp#L873
taskdata->td_flags.proxy = TASK_PROXY;

taskdata cannot be safely accessed after this line.

__kmp_task_finish () "passes ownership of taskdata" with this line.

At this point, __kmp_fulfill_event might release the task at any time.

Joseph sees a spurious assertion failure in
https://github.com/llvm/llvm-project/blob/master/openmp/runtime/src/kmp_tasking.cpp#L710

This is, when
https://github.com/llvm/llvm-project/blob/master/openmp/runtime/src/kmp_tasking.cpp#L906
is not reached fast enough.

Can the accesses be safely moved up?

Best
Joachim

Thanks Joachim for looking into this. For the record, the patch you posted at https://reviews.llvm.org/D79702 seems to fix the race for me. I was able to run several different configurations and the issues to be gone.

Cheers
Joseph