Currently, some CMake targets are excluded from libMLIR.so, e.g., test dialects, test runtimes, and ExecutionEngine. While I understand that some targets are undesired for releases, others might be interesting for downstream projects.
We want to migrate to using libMLIR.so for our CI builds to reduce memory pressure at link time. Unfortunately, we have an integration testing utility that depends on ExecutionEngine, and this thus requires static linkage against this library, which transitively pulls in a large portion of LLVM. Given that this integration test tool results in the highest linkage memory usage of all our executables, this is an issue.
Might it be acceptable to add the ExecutionEngine to libMLIR.so, or would it be more sensible to add a way to overrule the EXCLUDE_FROM_LIBMLIR CMake flag?
CCing @stephenneuendorffer, as he added this option to the ExecutionEngine CMake target.
I’m trying to remember the reasons behind this exclusion. I think it was that MLIR likes to only have a required dependency on the LLVM support library (which is logically a distinct project). The ExecutionEngine stuff is only in MLIR proper due to lack of a better home, and it too is logically a separate component (which pulls in a lot of LLVM).
My personal opinion is that the current approach to libMLIR is untenable: people who care build their own containing what they need, and those who just want to have something to dynamic link against are caught against these weird, half formed boundaries (ie. The component division is good but that shouldn’t imply static vs shared linking decisions).
I think that with the present project arrangement, the possibly best thing to do here is to create a shared library just for these runtime components. Other options:
- Use BUILD_SHARED_LIBS – with the usual caveats
- Show how to have people control and mix their own contents of libMLIR or create their own aggregate. (I think this is what you were getting at)
I’d rather not add another creeping dep here that we will have to undo later as the project organization matures.
Can we instead have a libMLIRExecutionEngine.so which links to libMLIR.so and libLLVM.so?
I’m not sure it’ll work because of libLLVMSupport, but maybe there could be an option for libMLIR.so to link to libLLVM.so for such case…
Just an aside: make sure you’re using lld, most of the memory pressure reports we see come from using the GNU BFD linker which is very inefficient from this point of view.
+1 - this is what I was thinking.
Thanks for the answers.
That seems to be the best and cleanest solution. I’ll investigate what’s necessary to force aforementioned linkage against both libMLIR.so and libLLVM.so, without breaking existing solutions.
It might take a while until I can tackle this issue, but when I do, I’ll make sure to keep you guys updated.
We already enforced this, as some of us experienced said issues years ago while working with GNU’s ld.