PSA: Python binding dependencies changing

MLIR
1

If you don’t maintain an out of tree project that uses the MLIR Python bindings as part of a larger super-project, you can stop reading.

There is work beginning to support nanobind for writing dialect extensions. Currently, the MLIR Python bindings are based on pybind11, but the intent was that this was always an implementation detail versus a hard coupling. Some of us are keen to go further and use nanobind throughout based on the extreme reductions in compile time/size and improved performance it brings – but that is a topic for a different day.

What this means for right now is that MLIR itself is about to grow a dependency on nanobind. This is needed to run the test suite, etc, and is not too different from other dependencies MLIR carries for completeness.

However, the way that MLIR and super-projects that use it collaborate on setting up Python development package dependencies has always been very fragile since there are multiple ways to instruct CMake to set up Python and pybind/nanobind/etc and the way that some of this stuff is distributed is not universally compatible. In the past, changes like this have caused a set of rolling issues across projects as folks integrate the patch and need to adapt build systems and CI infra in opaque ways.

To make this better for the future, I landed [mlir] Add option to disable MLIR Python dev package configuration. by stellaraccident · Pull Request #117934 · llvm/llvm-project · GitHub which adds an option MLIR_DISABLE_CONFIGURE_PYTHON_DEV_PACKAGES. If this is set, then the mlir_configure_python_dev_packages macro will no-op. In this case, the super-project is wholly responsible for providing the needed build dependencies, which is basically the equivalent of this today:

find_package(Python3 3.10 COMPONENTS Interpreter Development.Module REQUIRED)
find_package(pybind11)

If using the new nanobind support once that lands, that will also require an incantation as documented here.

It is always up to the super project to ensure that single, project wide dependencies are resolved properly. I’d recommend that if you manage such a project, that you set the above option and configure the components yourself so that you are future proof and have one easy place to change (in our project, we use FetchContent, but the best options require CMake versions that LLVM itself does not yet support and you are better off just owning this configuration downstream where you can set your own CMake version policy).

You can keep using it as-is if desired, but in the world we are heading to, the mlir_configure_python_dev_packages macro is really an implementation detail of MLIR itself, and using the above CMake option lets you insulate yourself from changes and conflicts that are going to cause churn there.

@makslevental, @jdd, @marbre

7 Likes
2

Thanks for tagging me Stella.

CIRCT’s Python module is built as a unified build (LLVM_EXTERNAL_PROJECT) so I don’t think it counts as a “super-project” from a Python/CMake perspective. For the most part, we use the standard mlir cmake functions (ie add_mlir_python_modules/declare_mlir_python_extension/etc.) so I think as long as your future changes don’t change the behavior of those cmake functions you won’t break us.

@mikeurbach Am I lying?

I do, however, have a runtime subproject in the CIRCT tree which doesn’t use them and finds pybind11 on its own (unless pybind11_DIR is set). (It doesn’t just use CIRCT/MLIR python cmake stuff since I want it to be build-able outside of the CIRCT code tree.) It’s not enabled in any of our CI gates so it could break silently. MLIR’s future changes might break it so I’d appreciate if you tagged me on anything that might break it.

Nanobind looks neat! I wasn’t aware of it. We already write custom C++ code for pybind11, so philosophically we’re already there.

3

Yeah, I haven’t looked at the build for CIRCT in quite some time. My advice on “super-project” was perhaps imprecise.

Really, what I would suggest is that if you have a cmake scope that is outside of LLVM/MLIR and has its own method for configuring python and/or configuring pybind11 or nanobind, then you should use the new cmake option at the outermost scope where that is done and just take control completely. That way you won’t be fighting MLIR’s built in heuristic for control.

There are a number of cmake quirks with both pybind and nanobind, and rather than listing them, I’ll just share my conclusion: it’s a losing battle having two parts of a project try to collaborate on providing the deps. Unfortunately, the quirks are based on both cmake version and whether you got pybind/nanobind from various kinds of source vs installed arrangements. Better to just not go there and take the easy way out by telling the innermost to not try.

There will need to be a real RFC for a more wholesale switch to nanobind. This PSA was just the result of a contributor sending a patch to enable nanobind support in out of tree dialects (and tested via the standalone example project), and I decided to help them out and clear some of the roadblocks to getting it landed.

Also, the newer versions of cmake fix a lot of things in this area. As LLVM moves forward, we can probably simplify the mechanism used for this stuff. That always comes with unpredictable issues. For my projects, we just chose to control our destiny on that stuff vs leaving part of the config logic in MLIR.

1 Like
4

How is the status of upstreaming CIRCT into the llvm-project monorepo? The thread RFC: Graduate CIRCT to monorepo? is dead.

CIRCT in the monorepo could make lots of things easier.

5

Off topic.

(Easier for CIRCT maybe. But then there is also a more than fair chance that the contributor would not have been interested in making these kinds of changes if needing to also fix everyone else’s project (and python bindings would have likely moved out of tree vs being upgraded))

6

That’s my definition of easier! :wink:

7

I understand. But if the only way to use this stuff is for everyone to join the monorepo and enforce a reciprocal duty to fix everything, all the time, we’d be better off calling this a failed project – because who wouldn’t want that deal, who decides, and who does the work?

On these threads, I’m just trying to split the difference between helping a contributor who has volunteered to make a long overdue improvement while doing what I can to smooth the path and over communicate so folks who have built on this stuff aren’t left in the lurch. I can think of many ways that this could be smoother and a better experience for library consumers, but that’s not how LLVM works… And I’m trying to keep those conversations “over there” so that the folks doing the work can keep going. I know I was terse with my comment. It wasn’t a lack of empathy for you guys.

2 Likes
8

I’m looking now, but we will need to do something. Either taking control like Stella suggested or making sure we’re aligned to the new way of discovering nanobind upstream. I’ll start looking at the immediate changes for CIRCT, as well as switching to nanobind in general.