Behavior of OMP_PROC_BIND=true

Runtimes OpenMP
#1

I am writing to figure out what is the expected result from get_omp_proc_bind when the environmental variable OMP_PROC_BIND is true. Based on https://www.openmp.org/spec-html/5.0/openmpse52.html I was expecting the result to be 1, which is what I get when using g++ version 9.2, but for clang++ I get 4, which is the same as spread. Any help would be appreciated since I am trying to match up results from an app compiled with clang and a performance model that currently returns 1 for get_omp_proc_bind when OMP_PROC_BIND=true.

-Drew

#2

I can’t find the spec text on the behavior of true but https://docs.nersc.gov/jobs/affinity/ says “Thread affinity is enabled with an implementation-defined default place list.”

In any case, you can interpret the return value of omp_get_proc_bind using the following (https://www.openmp.org/spec-html/5.0/openmpsu132.html):

typedef enum omp_proc_bind_t {
omp_proc_bind_false = 0,
omp_proc_bind_true = 1,
omp_proc_bind_master = 2,
omp_proc_bind_close = 3,
omp_proc_bind_spread = 4
} omp_proc_bind_t;

#3

I found where my confusion was coming from, but I would like clarification that omp_get_proc_bind is doing the right thing:

At line 5473 in kmp_settings.cpp

} else if (__kmp_nested_proc_bind.bind_types[0] == proc_bind_true) {
// OMP_PROC_BIND=true maps to OMP_PROC_BIND=spread.
__kmp_nested_proc_bind.bind_types[0] = proc_bind_spread;
}

Even if libomp uses spread internally for the case of OMP_PROC_BIND=true shouldn’t omp_get_proc_bind still report that the value true was provided instead of the value for spread?

-Drew

#4

I found where my confusion was coming from, but I would like clarification that omp_get_proc_bind is doing the right thing:

At line 5473 in kmp_settings.cpp

     } else if (__kmp_nested_proc_bind.bind_types[0] == proc_bind_true) {
       // OMP_PROC_BIND=true maps to OMP_PROC_BIND=spread.
       __kmp_nested_proc_bind.bind_types[0] = proc_bind_spread;
     }

Even if libomp uses spread internally for the case of OMP_PROC_BIND=true shouldn’t omp_get_proc_bind still report that the value true was provided instead of the value for spread?

I can see that we might want to do that but is it really more helpful?

~ Johannes

#5

The specification requires omp_get_proc_bind() to return the thread affinity policy to be used for parallel regions.
Though TRUE is among allowed return values, it is actually not a policy but rather enabling of some policy the runtime applies to threads binding.
So returning SPREAD looks more reasonable to me in this case. If we would return TRUE users will never know what policy the runtime is actually using.

Returning TRUE means for me:
- implementation wants to apply some thread affinity policy, other than standard MASTER, SPREAD or CLOSE;
- implementation does not want to bother with inventing a name for the policy;
- and/or implementation does not want to show any details of the policy to users.

I don't think that hiding details from user is a good behavior.
Though it may be possible that gcc has documentation with detailed description of the runtime behavior for OMP_PROC_BIND=true, I haven't checked.
Once we return SPREAD - the standard policy, we don't need to add any extra documentation.

-- Andrey

#6

Andrey,

How do you interpret the following?

"The behavior of the program is implementation defined if the value in the OMP_PROC_BIND environment variable is not true, false, or a comma separated list of master, close, or spread. The behavior is also implementation defined if an initial thread cannot be bound to the first place in the OpenMP place list.”

My issue is that I have a fake omp library that needs to return the same value as a real omp library from functions like omp_get_proc_bind, omp_get_max_threads, and omp_get_num_places if it is invoked with the same environmental variables. When implementing this fake omp, I interpreted the above quote to mean that if the OMP_PROC_BIND variable is set, then omp_get_proc_bind should return the value that corresponds to what the user requested.

My question boils down to: When OMP_PROC_BIND is used is the return value of omp_get_proc_bind allowed to be implementation defined or not?

-Drew

#7

From the spec, for omp_get_proc_bind: "The effect of this routine is to return the value of the first element of the bind-var ICV of the current task."

OMP_PROC_BIND value is used to determine how to set bind-var ICV. At some point, "true" has to be interpreted into an actual strategy, and that is what is stored in the ICV.

#8

Drew,

I agree with Terry, but would like to add some more details.

The statement you sited relates to the case when the value of OMP_PROC_BIND is not among cases listed in OpenMP specification as allowed - true, false, master, spread, close.
E.g. OMP_PROC_BIND=123456789. In this case the behavior is implementation defined. That means implementation can do whatever it wants, only requirement is the implementation needs to describe its actions in documentation. The statement has nothing with correct values of the OMP_PROC_BIND.

a fake omp library that needs to return the same value as a real omp library

What real omp library do you mean? There are many real omp libraries on the market - Microsoft omp library, IBM omp library, GNU omp library, etc. Each real omp library can have different behavior, so you need to pick one to emulate.

if the OMP_PROC_BIND variable is set, then omp_get_proc_bind should return the value that corresponds to what the user requested.

That's fine. But what if user requested to return implementation defined value? Like with OMP_PROC_BIND=true the implementation should return the first element on bind-var ICV list, which is implementation defined.

When OMP_PROC_BIND is used is the return value of omp_get_proc_bind allowed to be implementation defined or not?

My interpretation is: yes. Once the specification defined only three thread affinity policies - master, close and spread, - but also allows to specify that binding if just "enabled" via OMP_PROC_BIND=true, it is implementation defined how the bind-var list is filled in this case. Note that "true" is not a thread affinity policy, - that is my understanding.

Also, an implementation is allowed to add more thread affinity policies to the list of allowed ones. Then it is possible to set other policies via OMP_PROC_BIND, and retrieve via omp_get_proc_bind(). This of cause should be documented, and omp.h header should have all the definitions.

-- Andrey

#9

Drew,

I agree with Terry, but would like to add some more details.

The statement you sited relates to the case when the value of OMP_PROC_BIND is not among cases listed in OpenMP specification as allowed - true, false, master, spread, close.
E.g. OMP_PROC_BIND=123456789. In this case the behavior is implementation defined. That means implementation can do whatever it wants, only requirement is the implementation needs to describe its actions in documentation. The statement has nothing with correct values of the OMP_PROC_BIND.

a fake omp library that needs to return the same value as a real omp library

What real omp library do you mean? There are many real omp libraries on the market - Microsoft omp library, IBM omp library, GNU omp library, etc. Each real omp library can have different behavior, so you need to pick one to emulate.

I agree with you at this point. As long as everyone is in agreement that the result of omp_get_proc_bind is implementation defined, I will have to pick which omp library I want to emulate.

Thanks for the discussion.

#10

Hi all,

we discussed this in the OpenMP language committee. 5.1 will contain a
clarification that the thread affinity policy is implementation defined
in case of OMP_PROC_BIND=true.
Initializing bind-var with any of the predefined policies is one way to
document the choice transparently.
If omp_get_proc_bind returns proc_bind_true, the thread affinity policy
is implementation defined.

As Andrey pointed out, implementation defined means

that implementation can do whatever it wants, only requirement is the

implementation needs to describe its actions in documentation.

Although earlier versions of 5.1 did not contain this clarification,
this still reflects the intention.

Best
Joachim