[RFC] A reference pass-plugin in LLVM

Pass-plugins like Polly, Enzyme, Clad and omvll have a long history in LLVM. They allow third-parties to develop domain- and platform-specific extensions for their toolchains and they mitigate the trend to upstream everything into the monorepo. The New Pass Manager has a well-defined plugin interface and supports IR-level plugins today in the optimization pipeline. LLVM’s codegen pipeline doesn’t use the New Pass Manager (yet), so it’s out of scope for now.

In the last years I worked with various companies that develop pass-plugins for both, products and internal tools. We have been facing a number of challenges that ultimately indicate a lack of infrastructure for pass-plugins in upstream LLVM.

This RFC aims to improve our collective experience when building and distributing pass-plugins for LLVM-based tools. The idea is to introduce a reference pass-plugin that gets built and tested on a broad range of platforms and configurations. Ideally it engages toolchain vendors to include it in their distributions. I prepared the following patch as a baseline for a conversation:

Given that we find agreement on something around these lines, we could move on like this:

  1. Land the reference plugin and cross-project tests that reflect the status quo

  2. Roll out tests to cover a broad range of platforms and configurations

  3. Fix bugs and track new ones that we uncover on the way

  4. Extend the plugin interface for commonly requested features

Naturally, the earlier steps are quite sharp already and the later ones are still more blury. Input from the community is very welcome and will hopefully guide us through the process. Earlier this year, I gave a presentaiton and ran two round-tables on EuroLLVM. If you are looking for some more context, please find the recording here: https://www.youtube.com/watch?v=pHfYFGVFczs

Why a reference pass-plugin?

The infrastructure for pass-plugins has issues that have no visibility right now. This is due to missing test coverage and missing test execution:

What we are missing are end-to-end tests in LLVM, Clang and LLD, that run regularly on many bots. Adding a reference plugin under tools helps here, because it would be available by default and visible to subprojects. The tests in the initial patch cover a wide range of features accross subprojects and they are easy to extend.

Another aspect affects distributions: Pass-plugins use the LLVM C++ ABI, which comes with a lot of pitfalls that are difficult to debug. If toolchain distributions shipped a working reference plugin, then plugin vendors could compare their own implementations against it. Having an interesting reference plugin in LLVM, might incentivize toolchain vendors to ship it. This would help plugin vendors to build and maintain their products.

Why Python?

The initial patch proposes a generic pass-plugin that runs a Python script for the actual IR transformations, because:

  • Python is popular and there are various bindings to LLVM

  • Python is required to build and test LLVM, so it adds no additional dependencies

  • Python scripting helps us test a wide range of plugin features

  • Python’s Limited C-API is pretty stable, so we expect low maintenance and no additional compatibility requirements

  • Users can experiment with IR transformations without building LLVM, which may guide them on the plugin path rather than the upstreaming path (see the Realtime Sanitizer example in my EuroLLVM presentation)

  • A plugin that is useful might actually get shipped by toolchain vendors

  • There are (at least) two independent experimental implementations in the wild:

  • MLIR recently introduced a way to write passes in Python (when driving MLIR from Python): https://github.com/llvm/llvm-project/pull/156000

Why not roll out tests immediately?

Adding coverage for existing untested functionality isn’t trivial. It often uncovers platform-specific issues that need time to investigate. In order to keep the revert/reland churn low, it seems reasonable to build baseline tests for a restricted set of platforms. Once they are stable, we can lift restrictions step by step to extend coverage. This keeps diffs in reverts small and the number of affected bots and developers low.

The initial patch proposes the requires clause: native, system-linux, llvm-dylib, plugins, pypass-plugin. Tests should run on at least two build bots: llvm-nvptx-nvidia-ubuntu, sanitizer-x86_64-linux-android

What bugfixing is expected?

The initial patch outlines a few known bugs that should be fixed. The most famous one might be plugin-parameter parsing in Clang, which still requires an extra -Xclang -load -Xclang <path> today. This won’t be easy, so it’s best to nail down the status quo and get the big picture first.

We will likely find more issues on the way. We can track them with XFAILing tests and fix them where stakeholders see priority. Everything is better than the current situation where bugs are in the dark.

What are commonly requested features?

Real-world pass-plugins rarely run context-free static transformations. On the contrary, they often need additional context at build-time and support libraries at runtime. They also have their own release cycles and need to match customer toolchain versions. This entails requirements that we don’t currently meet, e.g.:

  1. Can we enable pass-plugins to inject extra cc1-options like -record-command-line? Command-line options provide insights like opt-level, hardening-mode or exception-model. These are essential for domain-specific transformations and caching of intermediate results (just like ccache for objects).

  2. Can we establish a feature for pass-plugins to add options/inputs on the link-line? This is currently possible with platform-specifc workarounds. A unified solution with tests would be very much appreciated.

  3. Can we make it easier to locate pass-plugins, so that -fpass-plugin=omvll loads /usr/lib/llvm-22/plugins/omvll-pluign.so? Tools with plugin support typically offer 3rd-parties a location to install their plugins. It makes loading easier and allows for implicit version match. It should be easy to come up with a sound traversal strategy.

I am sure this isn’t the end of the line, but for now, I think it’s enough planning to decide on the general direction. Looking forward for your feedback!

3 Likes

So this RFC also involves writing and landing a set of Python bindings?

I was aiming for a minimal explicit interface that won’t pull in more API calls than necessary. That’s why the plugin enumerates the few Python C-API calls that it uses explicitly. They should all be from limited stable API.

It seems like this proposal conflates two things: Testing of pass plugins, and shipping a “useful” pass plugin. I don’t think that these two things should be combined.

If the proposal were just to improve pass plugin testing, I’d say that’s an obvious “yes”. If the current testing is insufficient and we can improve it, we should. I wouldn’t expect such testing to involve Python though.

Having an interesting reference plugin in LLVM, might incentivize toolchain vendors to ship it.

But something that is intended for testing should explicitly not be shipped by toolchain vendors. The build system should not even install it.

If you want to introduce an interesting/useful pass plugin, which is intended for distribution by vendors, I think that’s a separate thing and needs to be able to stand on its own merits. That should be an RFC on the specific functionality of that plugin (including justification for why it is a plugin – though I guess linking to Python would be sufficient justification for that).

If you want to introduce an interesting/useful pass plugin […]

Thanks for your feedback. Well, the main goal is to introduce cross-project tests for existing functionality. That on its own is probably not worth a RFC, we could just make PRs. This was meant more like a heads-up, if there is agreement to make something useful on the way.

That should be an RFC on the specific functionality of that plugin (including justification for why it is a plugin – though I guess linking to Python would be sufficient justification for that).

Yes, more elaboration: The specific functionality is the Python interface. Users can connect their local Python to the opt pipeline of their production compiler and write passes without building LLVM. This is great for prototyping, instrumentation and experimentation.

LLVM has no Python bindings (anymore) upstream, but there are bindings out-of-tree that I mentioned. And yes, Python bindings only reach the C-API, which is not super rich, but this is a chicken-egg problem as well.

If the current testing is insufficient and we can improve it, we should. I wouldn’t expect such testing to involve Python though.

The Python integration is super useful for testing, because it’s an easy way to switch code paths on and off, e.g. which entry-points to trigger. Yes, we could do that with a C++ hard-coded plugin, but I am not sure it will get a lot simpler. MLIR uses a lot more Python scripts in their test cases and it seems to work well.

But something that is intended for testing should explicitly not be shipped by toolchain vendors. The build system should not even install it.

We do a lot of testing with llvm-objdump and we ship it in distributions. Why would we strictly avoid that?

Yeah, in the end the distribution aspect is a bit of a nice side-effect. If there is no agreement that a Python pass-plugin is something worth distributing, I can just cross that out.

Sorry, my first reply was “No, not landing Python <> C++ bindings like pybind or nanopy”, but you referred to Python <> LLVM bindings. I think the two out-of-tree bindings llvmlite and llvmcpy I mentioned are good and popular. No need to have our own ones in-tree. If we focus on something, the LLVM C-API would be a good candidate :slight_smile:

This sounds like an excellent proposal. We are frequently working on an LLVM plugin and have faced several of the issues ourselves, primarily around the mentioned bugs (like -load for plugin parameters).

I think the plugin infrastructure is currently at a point where it is powerful enough to be considered by projects, but has too many more minor issues to really be useful (e.g., having to effectively define the full plugin path for -fpass-plugin, not being able to inject compiler args, etc.).

I think that is a perfect first step.

1 Like

I don’t think depending on external Python bindings is viable. Not just because it adds another dependency to build LLVM, build also because if we make a change to LLVM (e.g. replace a deprecated API function), we cannot update that reference pass or its tests until these external bindings have been updated.

404

I don’t think depending on external Python bindings is viable.

Yes exactly. We don’t need bindings in-tree. The plugin exposes C-API refs and users can choose their favorite bindings. Here is an example from the PoC with llvmlite:

The interface was different here. The current proposal uses a run() function with module and context as arguments. If we wanted to build tooling on top of the plugin in-tree, then we would need a new set of bindings in-tree as well, but that’s a different story. We don’t need LLVM <> Python bindings for cross-project tests of the pass-plugin interface.

404

Thanks fixed: GitHub - weliveindetail/llvm-py-pass

In the OP you state:

So we need a Python binding, even if it is only for regression-testing whether that reference plugin actually works. AFAIU, the entire point of this RFC seems to be ensuring that the plugin infrastructure receives better testing. Can you clarify?

AFAIU, the entire point of this RFC seems to be ensuring that the plugin infrastructure receives better testing. Can you clarify?

Sure. The infrastructure is the glue that integrates the plugin into the running tool. Basically, everything up to the point where the proposed plugin hands off control to the run() function in Python.

Examples are the broken plugin-parameter parsing in Clang here and missing callbacks in some pipeline variants here. Right now it isn’t easy to tell, whether these are bugs, expected behavior or user error. For example: Is it expected that ThinLTO exercises vectorizer callbacks in LLD but not in Clang? Better testing gives us more transparency for such cases.

Python is a handy scripting layer that helps implementing these tests. We can use it to make sure the plugin has access to all the things we expose. Out-of-tree tooling with pass-plugin support like Rust or Swift will profit from that too: It will be easier for them to test whether they support the equivalent set of features!

All that in-tree Python bindings could contribute is testing of the C-API right? I think our unittests do a “good enough” job there and it’s not in scope for this RFC.

So we need a Python binding, even if it is only for regression-testing whether that reference plugin actually works.

We know that Python bindings exist out-of-tree that are actively maintained and work with regular Python distributions. So it should be sufficient for our regression tests to check, that the plugin provides a regular execution environment for Python.

The tests in the initial patch verify that the plugin successfully loads scripts that can:

  • load packages from the selected Python installation here
  • load packages relative to the file path of the selected script here
  • run Python commands that dump the given arguments and system information

I am happy to add check for further features. On example that comes to mind is correctness of implicit environment definitions like __file__. If you have more ideas, please let me know. I think the PR is a good place to discuss such implementation details.

I’ve started reviewing several aspects of the PR companion of this RFC. As someone with a decent experience with Native Python extension (author of pythran ) I’m having a good time reviewing this.

The technical choice of not requiring an abstraction like pybind11 or nanobind implies a deeper understanding of the Python C API (including PyObject reference management), but it removes any third-party dependencies.

Relying on dynamic loading instead of dynamic linking for libpython, combined with usage of the Limited C API , offers good portability of the generated plugin across Python version without the need of recompiling the Python plugin.

I’m not a big fan of the current CLI where, if my understanding is correct, the Python pass loader itself is a pass plugin with takes python scripts as options. I’d rather have the Python pass loader be part of the core and have the actual scripts be the plugins, and let the runtime use the right loader depending on the plugin loaded (.so → native loader, .py → python loader etc).

1 Like

Thanks for your feedback and reviews. I am glad you like it! Right, maximum compatibility across Python versions and no 3rd-party dependencies are important goals.

I’d rather have the Python pass loader be part of the core and have the actual scripts be the plugins, and let the runtime use the right loader depending on the plugin loaded (.so → native loader, .py → python loader etc).

This would affect the CLIs of all tools with pass-plugin support. I am not sure such tight integration is everybody’s cup of tea. As long as this is a plugin, everybody is free to opt-out. And because this is a plugin, we can simply load it to Rust to check their integration of pass-plugin features!

Is there a historical reason for this? If I read this file correctly, we don’t have any builders that enable examples and run check-llvm and check-clang. (NB: the config should enable both LLVM_BUILD_EXAMPLES and LLVM_INCLUDE_EXAMPLES.) The only CI pipeline that seems to have examples enabled is the pre-commit CI…

I think it would be good if at least some buildbots would do this. I think plugin testing requires LLVM_LINK_LLVM_DYLIB (not sure, maybe not)? So perhaps we could change one or two LLVM+Clang builders to link against the dylib and enable examples there so that the plugin infrastructure is also tested post-commit? (I have ~zero idea how our CI actually works.)

Is there a historical reason for this?

Good question! There might be more reasons, but AFAIK LLVM_BUILD_EXAMPLES has always been opt-in and we had no regression tests for them. Lang added tests for Kaleidoscope in the course of API changes for OrcJIT in 2015, because Kaleidoscope was broken many times (like this) and we had not noticed. These tests broke from time to time. I can imagine that the few buildbots that had examples enabled withdrew at that time.

I think it would be good if at least some buildbots would do this.

That would definitely help, but it doesn’t give us good coverage of platforms and configurations. It has always been the edge-cases that cause trouble: LTO, ThinLTO, BUILD_SHARED, LLVM_DYLIB=Off, Windows (native, mingw, cygwin). And I am not sure who would be willing to fight for it (i.e become maintainer for examples as a whole).

It would be good if our examples work, but in general I think testing for essential features shouldn’t depend on examples.

I think plugin testing requires LLVM_LINK_LLVM_DYLIB (not sure, maybe not)?

Plugins require -fPIC. LLVM_LINK_LLVM_DYLIB guarantees that they work, but it is not strictly necessary.

Thanks for the context.

Hmm, I disagree and just put up a PR to build examples that are used for tests by default; this adds 17 CUs and 24 tests. Given the size of LLVM, I think that’s tolerable to ensure that plugins work and the examples (i.e., what we expect our first-time users to look at and try out) actually work on more than the pre-commit CI environment. IMHO there’s not much of a point in having tests if we have no post-commit builders that run them and don’t make sure that they work on the platforms we consider important…

Sure, please go ahead. There is nothing we disagree about here. I think so far the problem was rather that no-one stepped up and spent the time maintaining them.

The PR’s regression test does the following

def run(mod, ctx, stage):
    print(f"0x{mod:016x} {stage}")

Strictly speaking, this only tests weather some pointer is passes (and doesn’t test ctx) at all, not that there is a working LLVMModuleRef behind it. This might be an acceptable compromise to not depend any LLVM bindings, but that should be documented. Overall, I also expect documentation or a How-To on its usage. The current PR only adds documentation for LLVM_PLUGINS_INSTALL_DIR.

I share this perspective, and look forward to any improvements in this area. The “pass” is one of the more stable concepts in LLVM, new pass manager migration notwithstanding, so it is one of the most reasonable extension points that LLVM could expose. I think it’s fair to say that LLVM’s default pass pipeline is geared towards optimizing C/C++ application programming, and it’s natural for downstreams to want to extend and adapt the pass pipeline for different applications.

However, this proposal is mostly focused on how to do this from Python, and my only input on that is, please do it in a way that doesn’t impose new API stability requirements, and I think you’ve done that successfully by limiting new APIs and focusing on the existing C API.

This is the main reason I’ve paused work on the dylib-by-default proposal. I should provide an update on that.

Let me summarize results so far.

There is no consensus to add the proposed plugin to LLVM at this point. I had seen it as a win-win situation to have a single entity for both, internal testing and as a tool for users. However, reviewers are not convinced and they have their reasons.

We have no Python bindings in-tree. We know there are working bindings out-of-tree, but we cannot use them for tests in LLVM, because they move forward in release versions. During release cycles such tests would be unfixable. Thus, we cannot exercise Python for testing beyond simple plugin interface checks. That makes the test story less attractive.

Example-based tests are considered sufficient. Alexis moved on with an effort to re-enable examples across build-bots in #171998’ff. Let’s see how this goes and what coverage we end up with.

We can keep using the Bye example for cross-project testing for now. However, first feedback indicates that extending Bye for tests might conflict with its use as an example.

There might still be room for a Python pass-plugin as a tool, but this could also live downstream in a standalone project or in one of the LLVM bindings projects.

I will leave this thread open for further opinions, follow-ups from third-parties and questions.