Help needed: Installing and Releasing Python-based MLIR Projects

libMLIRPublicAPI does not include most of the passes at the moment. But intrinsically exposing more public API reduces the chances for dead-symbols elimination.

Would -DBUILD_SHARED_LIBS=ON be enough or do you see a higher granularity division that wouldn’t match the CMake libraries?

LLVM releases every 6 months in general. The process is to branch of the main development branch, and then cherry-pick bug fixes until the release tester and release manager think they close all the release blocker issues. I don’t think there is anything more involved.

Possibly - should try this but was put off by the description of being for development. It will likely need an unclear amount of work for windows compat.

If we did finish off rough edges on this feature, maybe it would be fine…

Actually just found a 2014 talk on the release: slides / recording

My TZ isn’t usually great for the ODMs but I should be around tomorrow, badenh@cognitiviti.com

I managed to hack together the first step for python packaging (we discussed this ~a month ago): GitHub - stellaraccident/mlir-py-release (depends on https://reviews.llvm.org/D93936)

It uses some dirty hacks (relatively, not too bad, given that every setup.py is kind of a kludge), but should be relatively portable. I wired it up to also embed headers, shared libraries, and cmake export files, along with a little API for getting at the build settings. It needs some more work, but this should get us roughly to the point of being able to chain these projects together and build binaries. I’ll throw this in a github action shortly to do nightly binary builds on a couple of platforms and see where it goes.

I feel like once we add some more dialects/transforms to the ODS setup, it’s going to be a pretty good swiss army knife. The npcomp porting to use the new python API was pretty straight-forward, and the ODS integration was nice: https://github.com/llvm/mlir-npcomp/pull/144

Earlier this week I took the plunge into Python bindings land, so I will raise my hand as someone willing to help out with the Python packaging work. I’d also like to provide some feedback to everyone who has been working on the Python bindings and C API: this went really smoothly for me. Kudos for all the work that has gone into this so far; it’s really nice when someone can come along and things just work out of the box.

Thanks!! :slight_smile:

Thanks - it was a surprising amount of work to get it to this point, and there’s still plenty that needs to be done.

In terms of packaging, there is a big outstanding item mentioned in this thread: [llvm-dev] [RFC] Move py-mlir-release to new top-level repo in the LLVM org

At least in my view of the world, a fair bit of the rest is gated on finishing ⚙ D94387 Add new LLVMComponents CMake module. but that is likely going to take some time. An incremental step there might be to switch the release builds to -DBUILD_SHARED_LIBS=ON mode and separate the packages into an mlir-dev and mlir package. I also got some feedback from the Python releasing side about structuring it to build a source distribution that then builds into a binary wheel (right now it only builds a wheel). There is some subtlety to getting it right that is probably more conducive to a direct conversation (plus I’ve never built/deployed a python package of this complexity and am still learning). I’m going to keep spending the cycles I have on this working on the above diff for LLVMComponents, as without that, it gets very hard/impossible to build dependent projects.