[TCDG] Tensor Compiler Design Group Meeting notes 2025-09-17

MLIR Tensor Compiler
1

Hereby the meeting notes from this Wednesday’s Tensor Compiler Design Group meeting, September 17th.

17th September 2025

Attendees

@banach-space @sjarus @dcaballe @Groverkss @kuhar @qed @jpienaar

Topics

ConvertVectorToLLVM

Is it intended as the “always functional and optimal” lowering path from Vector to LLVM?

  • No. Historically, it was added as the very final step for lowering/converting from Vector to LLVM, i.e. a thin and relatively simple layer.
  • Over time, it has grown organically and has become much more powerful. It feels that we should revisit and re-define its scope.
  • While some users might be using it in production as the lowering pipeline from Vector to LLVM, this is not recommended. Folks present in the call build their own custom lowering pipelines. In general, we expect that other users should as well (to match the needs of their hardware).
  • To make users of ConvertVectorToLLVM more informed, we should remove all the default lowering paths (e.g., for vector.contract, vector.transpose) so that every user makes an optimal decision based on their needs. Alternatively, we can improve documentation.

Linearisation/Flattening and Unrolling

For context, see the discussion here: PR #151175.

Should “unrolling” be the de-facto legalisation path from Vector to LLVM + SPIR-V?

  1. Not all ops can be flattened (e.g., vector.transfer_read and vector.transfer_write), but all ops can be unrolled.
  2. Unrolling is not always the optimal choice, and often users will want to mix and match based on their targets.
  3. For some users, flattening is in fact sufficient (e.g. Diego).
  4. All in all, it feels that we should support customisation rather than enforce a specific path (i.e., treat “flattening” and “unrolling” as parallel techniques).

Requirements for New Vector Ops

  1. Request from Kunwar: we should document requirements for new Vector dialect ops, e.g., “For an op to support lowering to LLVM (or SPIR-V), it is required that the following passes are implemented.”
  2. This may effectively mean that we prioritise “unrolling” over “flattening.”

Action Items

  • Audit ConvertVectorToLLVM and remove defaults. Require users to customise the pipeline whenever more than one option exists. Alternatively, improve documentation.
  • Work towards customisation of the Vector lowering pipeline - users should have the necessary means to configure it as they need.
  • Improve the documentation for flattening/linearisation and unrolling - clarify the limitations and trade-offs. Users should be able to choose whichever works best for them.
  • Require all new Vector ops to implement unrolling.

Final points

@attendees, thank you for participating :folded_hands: Please leave comments if I missed or misinterpreted anything.

-Andrzej

3 Likes
2

This actually works well for CPUs. Please, keep it as the “naive” lowering, and let people overwrite with their own custom stuff. If we start carving functionality from MLIR, projects like the lighthouse will stop working (or have to carry local changes, which is against its charter).

1 Like
3
  1. Having default lowering path is more informative than just having an improved documentation. Defaults are the basic design goal of any compiler infra.
  2. Moreover, having a default lowering provides at least one base criterion for any further evaluation/exploration for optimal decision.
  3. Revisiting and defining the scope of vector-to-llvm without removing the default path make sense.
4

Always functional yes, otherwise the project is broken; optimal has never been in the charter. Targets are so wildly different they require applying different tools from the toolbox to get performance and the “virtual“ vector dialect is not the place to generally look for performace, let alone optimality (e.g. SIMD with/out hard performance 128B boundaries, GPUs, NPUs, g-TPU).

HW-specific vector dialects is where I would expect the performance-related features to go (e.g. transpose to specific SIMD lowerings in x86). Note the (very few) existing lowerings are already relying too much on LLVM peephole optimizations to my taste but they connected to concrete performance when implemented. The question is how much commonality and “massaging for the blackbox tool coming after” is required for perf.

+1 to renaming to NaiveConvertVectorToLLVM or something similar.

-1 to breaking functional paths; historically, I reluctantly added vector.shape_cast lowering in the past because of multiple requests to getting an e2e path working upstream. Marking it “naive” is the reasonable way to go.

Flattening + unrolling: +1 to getting good interfaces and coverage for all ops; the virtual vector dialect is a good place for such generic interfaces and transformation implementations. @cbate had added a first utility in PR #150K.

Also note that flattening or unrolling by itself is not sufficient IMO: one big question is how this materializes to asm (via an LLVM or SPIR-V blackbox). For instance “older” g-TPUs really like to manipulate large 2-D vectors in various ways (i.e. perform internal reshapes and transposes) whereas GPUs really need flattening to propagate unscathed to memory address computations (i.e. Triton’s strategy). I would therefore think we also need a specific transformation here.

By now we all understand LLVM’s peephole and backends work well when input is really massaged to what human writers are expected to write (e,g, x86 intrinsics which may or may not lower to shuffles and peephole). We do not have a similar general understanding of the SPIR-V bits.

One question I have for the experts here: is SPIR-V expected to be a “retargetable enough” abstraction for the targets it supports (tentatively and naively characterizing as “some GPUs and some CPUs”) or do we also expect to need different HW-specific abstractions to get the perf?

Also, where do SVE and SME stand on the flattening/unrolling/propagating to address computation spectrum? I would think a “convert to loop representation” for those would make sense but IANAE.

Orthogonally, it may be a good idea to bring in @ftynse’s normal form concepts to also revisit “blanket canonicalizations” vs “fixed-point towards a normal form”. A lot of “blanket canonicalizations” have been added over the years that need a gentle path out. Starting with renaming ConvertVectorToLLVM → NaiveConvertVectorToLLVM can be a good path towards moving canonicalizations to the NaiveConvert path and out of the general canonicalizations that would be inherited by the critical perf path. A subset of targets is always free to readopt a canonicalization pattern locally.

So overall I think I am seeing 4 transformations (flatten/unroll/“propagate”/“loop over”) ?

Glad to see progress on this front, thanks for pushing on this!

2 Likes
5

Thank you for all the feedback so far!

Just to clarify: ConvertVectorToLLVM is here to stay :slight_smile: We are not proposing to remove or trim it.
What I meant was: remove all the default settings. In other words, keep all the existing lowering code, but in cases where there are multiple lowering options – for example:

– we should not provide defaults. Instead, users should be required to explicitly pick.
(I realise this wasn’t clear in my previous post :sweat_smile:, sorry for the confusion!)

That raises two key questions:

  • What should the defaults actually be - how do we make that decision?
  • How do we ensure users are aware that these lowering paths can be configured?

Agreed - when a lowering succeeds, it should of course be correct. But do we guarantee that ConvertVectorToLLVM succeeds on all Vector code? That seems less clear.

:+1: from me as well.

This seems to be the most debated point. My view: we shouldn’t argue “unrolling vs flattening” as an either/or. Both have their use cases, so we need to support both.

That suggests we may also need a ConvertVectorToSPIRV?

From my perspective: we use both approaches depending on the situation. With scalable vectors, SVE and SME introduce unique challenges - e.g. shuffling is limited, and masking is often a bigger issue compared to other targets.

To summarise, proposed action items are:

  • Rename ConvertVectorToLLVM → NaiveConvertVectorToLLVM (please object if you disagree).
  • Remove default settings from ConvertVectorToLLVM / NaiveConvertVectorToLLVM.

Finally, I don’t see much discussion yet on this point:

My impression is that folks don’t view unrolling as “special” - but please chime in if you see it differently!

Thanks,
Andrzej

2 Likes
6

+1 to calling the pass naive to clarify the intent.

As a contrarian argument, not having any defaults make the system even less “usable” form an external point of view. One already has to assemble a pass pipeline, if one now has to configure passes in that pipeline as well, why even bother using something instead of just writing from scratch? :slight_smile:

For that, I’d need to present the idea to the broader audience first.

1 Like
7

Ready for reviews:

8

It can be done by knowing the host target or user specified target (DLTI or using the knowledge of build environment). Several flags can be set some default value based on the target/host. Other pattern related flags can be chosen based on common workloads.

9

When a default lowering is applied, emit a compiler log or diagnostic suggesting that the user can override it. Further, create some kind of “Lowering Strategy Guide” as part of documentation which compiler log can indicate during build. One challenge it throws is about verbosity which needs some mitigation.

10

This sounds like the right direction overall, but we don’t have any “compiler driver” infrastructure in-place that would drive this. We merely have one pass that “glues” patterns together :sweat_smile:

-Andrzej

11

IMO, Diagnostic would be internal to the pass and thrown based on the default value only of the respective flags.This way passes would be self sufficient independent of the “compiler driver”.