[RFC] Clarify the absence of API stability for the *device runtime* (aka. libomptarget-nvptx-sm_XX.bc)

As part of a patch to remove dead arguments [0] a discussion started which asked for an RFC [1].

Even if the API is stable, there is no guarantee for compatibility.

I was burnt by mixing old system libomp and newer libomptarget.

http://lists.llvm.org/pipermail/openmp-dev/2019-March/002439.html

As a user, I don’t want to get any headache with compatibility. The best way is not even trying it.
Unless developers are 100% sure mixing works and mixing is covered by rigorous tests, it is better to forbid users trying to mix and later run into problems.
Compile time stop is better than run time error.

In short, mixing Clang 10 with libomp/libomptarget 11 or mixing Clang 11 with libomp/libomptarget 10 is not expected by me.

Best,

Ye

Even if the API is stable, there is no guarantee for compatibility.
I was burnt by mixing old system libomp and newer libomptarget.
http://lists.llvm.org/pipermail/openmp-dev/2019-March/002439.html

As a user, I don't want to get any headache with compatibility. The best way is not even trying it.
Unless developers are 100% sure mixing works and mixing is covered by rigorous tests, it is better to forbid users trying to mix and later run into problems.
Compile time stop is better than run time error.

In short, mixing Clang 10 with libomp/libomptarget 11 or mixing Clang 11 with libomp/libomptarget 10 is not expected by me.

Best,
Ye

Ye Luo, Ph.D.
Computational Science Division & Leadership Computing Facility
Argonne National Laboratory

+1

Frankly, we're still at the stage where we're making all of this work for non-trivial production applications; we're not yet at the stage where we should be trying to provide cross-version stability on what is really a compiler-internal interface. We've not had a discussion, AFAIK, where we decided on a stability policy for this interface, it's an IR-level interface, and so our default should be to consider it version-locked to LLVM.

-Hal

I'll go ahead with the patches today.

What do you mean by fix the runtime part first?

~ Johannes

Sure :slight_smile: That is the proposal/patch.

Without backward-compatibility between libomp versions, how do we
ensure that the correct version is being linked? I fear a chaos on
user systems, e.g. after updating the linux distribution to a new
version of /usr/lib/libomp.so?

Although distributions may do library versioning (e.g. on Debian I see
libomp.so as a symlink to libomp.so.5), the clang driver just adds
-lomp or -lomptarget to link to the 'latest' version.

Michael

Without backward-compatibility between libomp versions, how do we
ensure that the correct version is being linked? I fear a chaos on
user systems, e.g. after updating the linux distribution to a new
version of /usr/lib/libomp.so?

Although distributions may do library versioning (e.g. on Debian I see
libomp.so as a symlink to libomp.so.5), the clang driver just adds
-lomp or -lomptarget to link to the 'latest' version.

Michael

We're not discussing these. You're talking about host-side runtime libraries. This is for the device-side runtime library only (e.g., libomptarget-nvptx-sm_XX.bc).

-Hal

Ye was referring to libomp/libomptarget specifically mixed with
different versions of clang. Could you clarify what "+1" to?

Michael

Ye was referring to libomp/libomptarget specifically mixed with
different versions of clang. Could you clarify what "+1" to?

Michael

Certainly. Good point. I was referring to the device-side runtime library (as in the title of the RFC), and I interpreted Ye's comments in that context. We do consider ABI stability for libomp. For libomptarget, on the host, it's not obvious to me that's really ready for ABI stability either, but as you point out, we might want to consider it as distributions are installing it in system-wide locations. That, however, should be a separate discussion (or, at least, a separately-titled discussion).

-Hal

Ye was referring to libomp/libomptarget specifically mixed with
different versions of clang. Could you clarify what "+1" to?

Michael

Certainly. Good point. I was referring to the device-side runtime library (as in the title of the RFC), and I interpreted Ye's comments in that context. We do consider ABI stability for libomp. For libomptarget, on the host, it's not obvious to me that's really ready for ABI stability either, but as you point out, we might want to consider it as distributions are installing it in system-wide locations. That, however, should be a separate discussion (or, at least, a separately-titled discussion).

-Hal

Without backward-compatibility between libomp versions, how do we
ensure that the correct version is being linked? I fear a chaos on
user systems, e.g. after updating the linux distribution to a new
version of /usr/lib/libomp.so?

Although distributions may do library versioning (e.g. on Debian I see
libomp.so as a symlink to libomp.so.5), the clang driver just adds
-lomp or -lomptarget to link to the 'latest' version.

Also, if we wanted to change clang so that it linked version-locked versions of these libraries, -lomptarget-11 or whatever, that, in my opinion, would also be a reasonable choice to discuss.

One thing worth capturing is the extent to which these things are connected. There is a relationship between libomptarget and its plugins, and the plugins and the device-side runtimes. There is an ABI boundary there somewhere. If we change nothing else, we might need to consider ABI stability of this part of the device-side interface.

-Hal

The official distribution apt.llvm.org contains libomptarget.so for
LLVM 7 to 10, but each into separate directories under
/usr/lib/llvm-<version>/libomptarget.so. The prebuilt binaries under
https://releases.llvm.org/download.html puts it directly under
<prefix>/libomptarget.so. If libomptarget is version-locked, users
need to be careful about pointing to the right LD_LIBRARY_PATH.
However, I could not find target device plugins in the distributions
(such as lib/libomptarget-nvptx-sm_60.bc when built on a machine with
CUDA). The official ubuntu repository doesn't contain libomptarget at
all. Arch Linux contains at least the x86_64 rtl
(https://www.archlinux.org/packages/extra/x86_64/openmp/files/)
without any versioning resolution.

Should make it explicit what the compatibility guarantees for
libomptarget are, maybe even discourage OS distributions to
pre-package libomptarget into ldconfig default paths? At least on Arch
Linux updating the openmp package will break previously
compiled-with-offloading binaries.

Michael

I asked the maintainer to install libomptarget and the plugin because
otherwise there are link errors when compiling an OpenMP application
with target directives. That's worse from a users perspective.

Really, this thread was about the device runtime which I still think
would be nice to keep working with one released version of Clang. Based
on the other comments in this thread, that's not going to happen -
acknowledged.
The host runtimes on the other side are linked dynamically and IMO it's
one of the strengths of libomp to mix-and-match versions. Unless
there's a compelling reason, I'll make another plea to keep
compatibility.

Jonas

Arch Linux contains at least the x86_64 rtl
(https://www.archlinux.org/packages/extra/x86_64/openmp/files/)
without any versioning resolution.

Should make it explicit what the compatibility guarantees for
libomptarget are, maybe even discourage OS distributions to
pre-package libomptarget into ldconfig default paths? At least on Arch
Linux updating the openmp package will break previously
compiled-with-offloading binaries.

I asked the maintainer to install libomptarget and the plugin because
otherwise there are link errors when compiling an OpenMP application
with target directives. That's worse from a users perspective.

Really, this thread was about the device runtime which I still think
would be nice to keep working with one released version of Clang. Based
on the other comments in this thread, that's not going to happen -
acknowledged.
The host runtimes on the other side are linked dynamically and IMO it's
one of the strengths of libomp to mix-and-match versions. Unless
there's a compelling reason, I'll make another plea to keep
compatibility.

Right, this thread was not about *any* `.so` runtime.

I asked the maintainer to install libomptarget and the plugin because
otherwise there are link errors when compiling an OpenMP application
with target directives. That's worse from a users perspective.

Isn't subtly miscompiling because a runtime function semantics changed
between versions worse than a link-time error?

Really, this thread was about the device runtime which I still think
would be nice to keep working with one released version of Clang. Based
on the other comments in this thread, that's not going to happen -
acknowledged.
The host runtimes on the other side are linked dynamically and IMO it's
one of the strengths of libomp to mix-and-match versions. Unless
there's a compelling reason, I'll make another plea to keep
compatibility.

I wonder whether we can cleanly separate libomptarget ABI stability
from device-side runtime ABI.

In the current clang driver, when the runtime .bc is found it is
linked into the executable using the -mlink-builtin-bitcode flag to
-cc1. If not found, it emits a warning:

No library '%0' found in the default clang lib directory or in LIBRARY_PATH. Expect degraded performance due to no inlining of runtime functions on target devices.

I am not sure what happens when not found. I assume it requires the
libomptarget-nvptx.a (containing the device-side runtime) to be linked
statically instead and the cuda runtime will link the device-side
functions (i.e. without inlining as by the warning).

The question is whether libomptarget-nvptx.a is considered part of the
distribution (i.e. installed next to libomptarget.so as `ninja
install` currently does). Maybe only libomptarget-nvptx.a needs to be
versioned (e.g. by appending a suffix such as
libomptarget-nvptx-<version>.a) to ensure that when linking, the
device code matching the version of the clang driver is linked
(theoretically, the driver could be used to link object files from
other versions of clang, but I think that risk is negligible).

As by a comment in the CMakeLists.txt, "Dynamic linking is not yet
supported by the CUDA toolchain on the device.", which inhibits device
RTL in the libomptarget.so itself. Also, the proposal by Greg Rodgers
for fat static libraries presented at LLVM-CTH
(https://hps.vi4io.org/events/2020/llvm) only considers static
linking.

Michael

Also, if we wanted to change clang so that it linked version-locked
versions of these libraries, -lomptarget-11 or whatever, that, in my
opinion, would also be a reasonable choice to discuss.

One thing worth capturing is the extent to which these things are
connected. There is a relationship between libomptarget and its plugins,
and the plugins and the device-side runtimes. There is an ABI boundary
there somewhere. If we change nothing else, we might need to consider
ABI stability of this part of the device-side interface.

The official distribution apt.llvm.org contains libomptarget.so for
LLVM 7 to 10, but each into separate directories under
/usr/lib/llvm-<version>/libomptarget.so. The prebuilt binaries under
https://releases.llvm.org/download.html puts it directly under
<prefix>/libomptarget.so. If libomptarget is version-locked, users
need to be careful about pointing to the right LD_LIBRARY_PATH.
However, I could not find target device plugins in the distributions
(such as lib/libomptarget-nvptx-sm_60.bc when built on a machine with
CUDA). The official ubuntu repository doesn't contain libomptarget at
all. Arch Linux contains at least the x86_64 rtl
(https://www.archlinux.org/packages/extra/x86_64/openmp/files/)
without any versioning resolution.

Should make it explicit what the compatibility guarantees for
libomptarget are, maybe even discourage OS distributions to
pre-package libomptarget into ldconfig default paths? At least on Arch
Linux updating the openmp package will break previously
compiled-with-offloading binaries.

You mean that it will break them *if* we make an ABI-breaking change in libomptarget. Changing the device-side runtime doesn't necessarily imply that. Nevertheless, certainly good to know.

-Hal

I asked the maintainer to install libomptarget and the plugin because
otherwise there are link errors when compiling an OpenMP application
with target directives. That's worse from a users perspective.

Isn't subtly miscompiling because a runtime function semantics changed
between versions worse than a link-time error?

Really, this thread was about the device runtime which I still think
would be nice to keep working with one released version of Clang. Based
on the other comments in this thread, that's not going to happen -
acknowledged.
The host runtimes on the other side are linked dynamically and IMO it's
one of the strengths of libomp to mix-and-match versions. Unless
there's a compelling reason, I'll make another plea to keep
compatibility.

I wonder whether we can cleanly separate libomptarget ABI stability
from device-side runtime ABI.

I think that part of the issue here is that the device-side runtime has two ABIs. One used for interaction with the plugin (how to receive kernel parameters, etc.). A second used to communicate with the application code on the device. I believe that the intent of this thread was only to talk about the second one. My understanding is these two ABIs can be considered separately. Johannes, is that correct.

-Hal

Correct.

I inferred from your response that we have no such guarantee yet, we just haven’t broken it yet.

However, I think that change will break ABI of libomptarget-nvptx.a

Just want to clarify what ABI stability guarantees we have.

Michael

Correct.

I inferred from your response that we have no such guarantee yet, we just haven't broken it yet.

However, I think that change will break ABI of libomptarget-nvptx.a

I think that this is an important point. I interpreted this thread in the context of the API between the runtime and the device-side application code. Not the ABI between the plugin and the device-side runtime. I suspect these two are separable, but we should definitely clarify this.

-Hal

I asked the maintainer to install libomptarget and the plugin because
otherwise there are link errors when compiling an OpenMP application
with target directives. That's worse from a users perspective.

Isn't subtly miscompiling because a runtime function semantics changed
between versions worse than a link-time error?

Really, this thread was about the device runtime which I still think
would be nice to keep working with one released version of Clang. Based
on the other comments in this thread, that's not going to happen -
acknowledged.
The host runtimes on the other side are linked dynamically and IMO it's
one of the strengths of libomp to mix-and-match versions. Unless
there's a compelling reason, I'll make another plea to keep
compatibility.

I wonder whether we can cleanly separate libomptarget ABI stability
from device-side runtime ABI.

I think that part of the issue here is that the device-side runtime has two ABIs. One used for interaction with the plugin (how to receive kernel parameters, etc.). A second used to communicate with the application code on the device. I believe that the intent of this thread was only to talk about the second one. My understanding is these two ABIs can be considered separately. Johannes, is that correct.

Correct.

Furthermore, the one we want to change now is neither backward nor forward compatible, as I mentioned in my initial email. At least assuming the compilation mode via `libomptarget-nvptx-<version>.a` is is something we actually do not support because even if it would work it should never be done.

Correct.

I inferred from your response that we have no such guarantee yet, we just haven't broken it yet.

However, I think that change will break ABI of libomptarget-nvptx.a

I think that this is an important point. I interpreted this thread in the context of the API between the runtime and the device-side application code. Not the ABI between the plugin and the device-side runtime. I suspect these two are separable, but we should definitely clarify this.

Neither should be considered stable at this point. We kept libomptarget stable while we recently added functions but I would not assume it will stay that way. As I mentioned before, for the device runtime there is basically no supported way today in which we could even guarantee stability.