I’ve been mulling on how to integrate SIMD acceleration into a compiler I’m working on and I’ve just been sitting in on a talk by the maintainer of Highway. MLIR has some support for SIMD, for example Arm Neon and SVE but highway supports a wide range of targets (notably RVV which is of great interest to me) using a unified API which makes it look very appealing at first glance. From a quick web search it seems that as of yet nobody has tried to integrate this with MLIR.
I could imagine that my MLIR-based compiler might lower portions of code to Vector (and maybe further to some custom Highway dialect), at the same time outputting (maybe via EmitC) a C++ header containing template instantiations for the necessary function templates from the Highway header. My compiler could then emit LLVM IR into which the Highway function definitions could ideally be inlined.
Now I’m wondering if this sounds like a reasonable undertaking or if there are points I haven’t considered that would make it much more sensible to e.g. flesh out the Arm dialects and/or provide a RVV dialect myself if I want to target these platforms in particular.
[Side note] If you’re thinking of integrating this into upstream MLIR, then we may need to wait for our project charter to be established before we are ready to discuss details:
I think this kind of project needs guidance, and for that, we need the charter and governance as @banach-space said.
The main thing here is that very often people come to MLIR and see that they can add a hardware dialect without realizing they don’t need to for 99.9% of the problems they want to solve.
So, if you have a problem, and you can’t solve with vector, arith, math, then we may need to improve those dialects instead of creating a new one.
But if you have an instruction that is exclusive to an architecture and it cannot be represented as an existing pattern, then maybe it’s worth adding an upstream dialect about it.
Do not take the existing dialects as an example. We’re discussing what to do with them and it’s not clear they need to stay where they are in their current shape.
There must be a higher bar to adding a new dialect upstream than as a side project or lack of awareness of what can/should already be done with existing dialects.
The main thing here is that very often people come to MLIR and see that they can add a hardware dialect without realizing they don’t need to for 99.9% of the problems they want to solve.
From my standpoint, I’m not actually sure if I could upstream this in the first place since it would intersect with work I’m doing for my employer. I was mainly interested in the comparison between portable SIMD libraries and SIMD dialects. But it’s good to know that a charter is underway, I’m going to follow that discussion in case the need to upstream something arises.
I would be very interested in knowing the actual gaps between targeting things with Vector vs Highway - have you considered making such comparison for one of the SIMD architectures targeted by Vector?
Not yet, I suppose my main motivation was RVV support. I have not investigated in detail how complete ARM support is vs Highway and/or how performance compares. That could actually be an interesting project. I was actually a bit surprised that there has been no collaboration in this direction even though both projects have been cooked up at Google.
We actually did have a discussion at Google But we didn’t pursue it at the time given where the groups were focussed (it was a 3 way discussion and the application team was interested in something else).
I think the main difficulty you’d run into is that Highway uses C++ templates and define macros. So you may end up wanting to introduce your own shim layer (say a header with all of these) to make EmitC side simple. Mixing kernels and compilation is quite natural (I recently dusted off the first slide deck we made about MLIR and we were talking about it there too). And this seems like it could be an interesting way.
Mmm, one could make a simple “splitter” so that one could even use the upstream execution engine and it would insert the required set of calls in the generated binary blob and emit files to be compiled and later linked in (reminds of the old CUDA splitting ). I mean one could go much more complex, but I’m assuming as you are working at this level it’s embedded in another system already, so this would provide convenient grouping mechanism for entry.
I’ve been looking into having a “micro-kernel” dialect that allows us to define function calls that have special meaning (ex. thread-local, no-side-effects, volatile) and have other passes like inliner and LICM to understand that and move them to the right place.
I think we need to look at this in a wider scope, trying to understand what we’re “calling” here. An IR function (a la Triton)? A fat binary that has all the functions? A JIT library that may fail at run time if called wrong (but won’t fail at compile time because compiler doesn’t know)? A template library that will be instanciated after the compiler has finished (ex. emitC)?
Would be awesome to have a very simple dialect for this. Perhaps just a few special function attributes? We’d need to teach existing transforms to work around it, too.
A quick response is that the upstream MLIR vector dialect supports a range as broad as Google Highway, including x86 AVX2/AVX512, ARM NEON, ARM SVE, and RVV.
Thanks to @banach-space for mentioning my previous work. Regarding the implementation of architecture-specific support and general performance portability solutions, we have had a series of discussions and efforts in recent years. @Time0o You can refer to some of the following topics:
I am currently conducting research on this. As it has not been published yet, I cannot disclose the actual experimental data at this point, but I can broadly say that kernel performance implemented using the MLIR vector dialect is on par with Google Highway and even better on the RVV platform when the fixed/scalable type feature is correctly utilized.
I think the motivation for emitting Google Highway kernels in an MLIR-based compiler is quite limited. From a performance perspective, using the vector dialect directly is sufficient. From an implementation standpoint, using vector dialect is easier to maintain than emitting Google Highway code.
My experience working on RVV at Google using the Vector dialect and the existing ecosystem was positive. We achieved high ROI by taking the VLS route and reusing the existing Linalg and Vector infrastructure, leaving VLA complexity to the LLVM backend.
The main challenge with introducing an RVV dialect upstream has been how to integrate it with the rest of the ecosystem. We had promising ideas but they lacked traction and prioritization. As mentioned before, this requires guidance and enough people backing it up and committing to maintain it.
For initial VLA experimentation in MLIR, consider using LLVM’s VP intrinsics. This approach should be relatively easy to have working and enough to generate custom VLA code for RVV in MLIR.
But we didn’t pursue it at the time given where the groups were focussed (it was a 3 way discussion and the application team was interested in something else).