We had some previous discussion about “unified builds”, where LLVM and CIRCT are configured in one CMake invocation, and compiled at once: Improve the Circt build system · Issue #208 · llvm/circt · GitHub. Following that, we have supported a unified build approach using
LLVM_EXTERNAL_PROJECTS. There are numerous benefits to this approach, as mentioned in the issue, which I have personally enjoyed.
To take advantage of the major improvements to the Python bindings being introduced in ⚙ D106520 Re-engineer MLIR python build support., we will be required to use unified builds, at least when building the CIRCT Python bindings. There was more discussion about this in Discord, for background.
I want to post here for visibility before just making a PR.
For people who are not building the Python bindings, I’m hoping to set this up so we can continue building CIRCT against a pre-built LLVM. Notably, that is how our CI is building CIRCT on each PR/commit to main, and I don’t want to change that (yet).
For people who are building the Python bindings, there are two approaches. For one, you can use
LLVM_EXTERNAL_PROJECTS, as mentioned in the original issue. This is how I’ve been building CIRCT, and it works with the upstream improvements. The other option is to continue building the same way, but have our
CMakeLists.txt include LLVM as a source-level dependency, rather than a library through
find_package. This is how NPComp is being updated here. This approach can be taken only when Python bindings are enabled, so those who don’t care may still depend on LLVM as a library.
Either way, a unified build means we can no longer cache a pre-built LLVM for CI or local development. I have been using CCache for local development, and this was suggested for CI as well. There is a GitHub action for it, which is being used in an MLIR-based project already. Since the Python bindings are only tested nightly, my plan is to first update the nightly tests to work with one of the two options above, and follow up with CCache to hopefully speed them up.
Some concrete questions:
- Is switching to a unified build a show-stopper of anyone using Python? I’d hope not, since this is how CIRCT will be built if it ever graduates from incubator into LLVM proper. CCache really helps for local development in this setup, if you aren’t using it already.
- For people building Python bindings, is the
LLVM_EXTERNAL_PROJECTSapproach preferred, or is there a desire to have the other approach work, like how NPComp does it? That will take a little more CMake tweaking, but it shouldn’t be a problem. Both can work, but I’d like to pick one to document, test in the nightly CI, etc. My preference is to use
LLVM_EXTERNAL_PROJECTS, since we already have two different ways to build CIRCT in
CMakeLists.txt, and if we take the other route, now we will have three.
- For people not building Python bindings, is there any desire to move towards a unified build for CIRCT? It’s been supported for a while, but it’s not what we document in the README, etc. I don’t want to force this yet, but I’m curious if there is any desire from the community to move this direction eventually. Perhaps after the Python setup is well tested, and we have a chance to test the CCache GitHub action, we could consider moving the documentation, CI, etc. to this approach.