PSA: MLIR distribution as python wheel

MLIR
1

In my neverending quest to bring the word of MLIR to as many people as possible, I’ve started building distros (i.e., ninja install) as python wheels: GitHub - makslevental/mlir-wheels. What does that mean? It means you can do this:

$ pip install mlir -f https://github.com/makslevental/mlir-wheels/releases/expanded_assets/latest

which will install a thing that will let you do this:

import mlir
print(mlir.__path__[0])
>>> /home/mlevental/mambaforge/envs/nelli/lib/python3.11/site-packages/mlir

where site-packages/mlir looks like this:

$ tree -L 1 site-packages/mlir
site-packages/mlir
├── bin
├── include
├── lib
├── python_packages
├── share
└── src

i.e., a full distribution of MLIR (and LLVM) that can be used to do out-of-tree builds i.e.,

-DCMAKE_PREFIX_PATH=$(python -c "print(__import__('mlir').__path__[0])")

What’s the point of packaging a primarily binary distribution as a python wheel?

  1. Using --extra-index-url or --find-links, you can include these wheels as a dependency in your requirements.txt, like so:
    -f https://github.com/makslevental/mlir-wheels/releases/expanded_assets/latest
mlir==17.0.0+6c7fd723
    i.e., releases/expanded_assets/latest works as a pypa-compatible package index.
  2. The wheels are built using cibuildwheel, so it’s guaranteed that the distribution is manylinux compatible, and thus can be used a foundation for other manylinux compatible python wheels.

Right now I’m building wheels for {(x86_64, aarch64/arm64/ARM64) ⨯ (linux, mac, windows)} - {(linux, aarch64), (windows, ARM64)}, i.e., arm builds of both linux and windows don’t work yet.
It’s on my todo list but both of these are vexing for their own distinct reasons so who knows :person_shrugging:.

The repo kicks off a new build (of TOM of llvm/llvm-project) at '00 01,07,13,19 * * *', i.e., four times a day. If you want some commit that’s not there just open an issue and I’ll kick off a build.

Note, there’s no possible future name collision here because you can’t pip install these from anywhere other than that releases page/index. So I’m not unintentionally (or intentionally) namespace squatting, but if someone thinks I should rename the package to something else, I’m not opposed.

:pray: to MS for funding my neverending quest (even if they’re not aware lol).

8 Likes
2

This is great. Should we find a more permanent place to push these wheels out ?

3

The natural place is llvm/llvm-project right? Honestly not sure because it’s above my paygrade, but I’m happy to colab with anyone who wants to help me with the politics of making them semi-official.

4

work like a charm! Many thanks for building this :heart:

1 Like