Exposing the New Pass Manager in the LLVM C API

Hi all,

At some time in the near future we would like to allow for Rust to use the new PM. As I understand it, Rust uses LLVM through the stable LLVM C API, which currently only provides an interface to the legacy PM. We wanted to ask what people think about how we should go about exposing the new PM through this API. We can think of 3 possibilities for now:

  1. Just replace the API to expose the new PM instead of the legacy one.

I imagine we wouldn’t want to do this since it would be a pretty sudden change and there wouldn’t be new PM equivalents for the Initialize/FinalizeFunctionPassManager functions.

  1. Add a new interface that exposes the new PM while still keeping the current one.

The interface would probably be similar to the current one but just not have the corresponding Initialize/FinalizeFunctionPassManager functions. I think with this method we could also slowly roll out usage of the new PM through the C API and eventually deprecate the legacy PM interface in favor of the new one.

  1. Add a build flag that switches between PMs used in the C API.

This could just be some flag in CMake that users can change at build time.

Any other suggestions/thoughts?

  • Leo

Hi all,

At some time in the near future we would like to allow for Rust to use the new PM. As I understand it, Rust uses LLVM through the stable LLVM C API, which currently only provides an interface to the legacy PM. We wanted to ask what people think about how we should go about exposing the new PM through this API. We can think of 3 possibilities for now:

Hi Leo,

I agree that increasing coverage of the stable C API (especially for such essentials as the new PM) is a worthwhile goal that will benefit various projects using LLVM.

About Rust specifically though, note that we’re not restricted to the C API. Whenever we need something that isn’t part of the C API (which is common), we just create our own bindings for it, i.e., write some C++ code that wraps whatever we need in a C API (see the src/rustllvm directory). It also seems unlikely Rust’s use cases can be fully served by a C API in the near term, we already have substantial amounts of custom C++ code interacting with the legacy PM to do things not available in the C API (e.g. driving ThinLTO) and this customization need won’t go away with the PM switch.

Regardless, more C API coverage is still nice for Rust, and will be a much bigger enabler for other projects that don’t have their own bindings to fill the gaps.

  1. Just replace the API to expose the new PM instead of the legacy one.

I imagine we wouldn’t want to do this since it would be a pretty sudden change and there wouldn’t be new PM equivalents for the Initialize/FinalizeFunctionPassManager functions.

  1. Add a new interface that exposes the new PM while still keeping the current one.

The interface would probably be similar to the current one but just not have the corresponding Initialize/FinalizeFunctionPassManager functions. I think with this method we could also slowly roll out usage of the new PM through the C API and eventually deprecate the legacy PM interface in favor of the new one.

+1 for this option.

  1. Add a build flag that switches between PMs used in the C API.

This could just be some flag in CMake that users can change at build time.

I don’t think this is a good option because many consumers of the C API don’t control the build process of the LLVM they use (e.g., they’re linked to a distro-provided library). It’s also not useful to programs that want to leave the choice of pass manager to runtime/end users, like clang does with -fexperimental-new-pass-manager.

Cheers,
Robin

At some time in the near future we would like to allow for Rust to use the new PM. As I understand it, Rust uses LLVM through the stable LLVM C API, which currently only provides an interface to the legacy PM. We wanted to ask what people think about how we should go about exposing the new PM through this API. We can think of 3 possibilities for now:

+1 to allowing the NewPM to be used from the C API.

The stability expectations on the C interface are higher than those on
the C++ interface. You should treat the C API as stable as much as
possible. This means your option (2) -- adding a new parallel API for
the NewPM, while keeping the old PM API -- is really the only _real_
option that is available here.

Cheers,
Nicolai

Does this mean that we will have to maintain two pass mangers forever?

Michael

This is a best effort stability, not a “guaranteed forever”. This is actually described here: https://llvm.org/docs/DeveloperPolicy.html#c-api-changes

Hi,

I agree that increasing coverage of the stable C API (especially for such
essentials as the new PM) is a worthwhile goal that will benefit various
projects using LLVM.

Same. For projects that need to compile against various versions of
LLVM, and that don't have very devoted resources, it's otherwise quite
hard to keep up.

It also seems unlikely Rust's use cases can be fully served by a C API
in the near term, we already have substantial amounts of custom C++
code interacting with the legacy PM to do things not available in the
C API (e.g. driving ThinLTO) and this customization need won't go away
with the PM switch.

It'd kinda be good to encapsule this logic better. There's quite a few
copies of thinlink like logic out there. I e.g. wrote my own crappy
version for postgres' JIT, Rust has a copy, etc. And it's really hard
to understand from the outside what one even iss supposed to use.

> 2) Add a new interface that exposes the new PM while still keeping the
> current one.
>
> The interface would probably be similar to the current one but just not
> have the corresponding Initialize/FinalizeFunctionPassManager functions. I
> think with this method we could also slowly roll out usage of the new PM
> through the C API and eventually deprecate the legacy PM interface in favor
> of the new one.
>

+1 for this option.

Same.

Greetings,

Andres Freund