[RFC] Allow pointers as element type of `vector`

MLIR
1

Context

In LLVM IR, the element type of a vector can be a pointer type. However, the MLIR VectorType does not support pointer element types. Instead, users must use a special LLVMVectorType that only exists because of this limitation of VectorType.

// !llvm.vec<2x!llvm.ptr>
<2 x ptr>

LLVM IR currently maps to mixture of vector types in MLIR: VectorType and LLVMVectorType are two of them.

Proposal

Allow pointer-like types as an element type of VectorType. Disallow !llvm.ptr as an element type of !llvm.vec. The goal is to:

  1. Have a single vector type used in the MLIR LLVM dialect that maps to the vector type in LLVM IR. This would simplify the usage of the LLVM dialect for downstream users. (This RFC can be a first step towards removing all custom vector types in the LLVM dialect.)
  2. Add support for vector<ptr> to certain vector dialect ops. For example, users may want to use vector.extract/insert to extract/insert from/into a vector of pointers instead of llvm.extractelement/insertelement because they provide additional folding opportunities. Vector dialect ops cannot use !llvm.vec.

Implementation Outline

The above PR adds a new type trait to work around layering limitations. The MLIR vector type is part of the IR build unit, which cannot depend on any dialect, in particular the LLVM and ptr dialects.

Know Limitations

Pointer types do not have a known bitwidth, so trying to compute the byte size of a vector of pointer will fail. The MLIR vector type already supports Index as an element type, so users of the vector type should already be aware of the fact that it is not always possible to compute a vector’s byte size.

Alternatives

Instead of making VectorType closer to the LLVM vector type, the LLVM dialect could choose to abandon the usage of the MLIR vector type and use !llvm.vec in all places where vector is currently used. This would be another step towards Goal 1.

2

If data layout was mandatory, like in LLVM IR, it would have.

3

-1 to making the DL mandatory in all circumstances, as not all IRs require one. However, +1 on removing all related bitsize methods from types and forcing users to have a DL to query any type size related info.

4

I’d like to say I’m not happy with special-casing pointers here. I’d like to ((once again, I think) propose “open vector” by way of a ScalarTypeInterface.

ScalarTypeInterface is

/// Return the bitwidth that this type always has by its definition 
/// or std::nullopt if there is none or this width is context-dependent,
//// as with index or pointers.
std::optional<int64_t> getInherentBitwidth();

/// Look up the bitwidth of this type in a `DataLayout`,
/// returning `std::nullopt` if the lookup fails. If getInherentBitwidth()
/// is defined, this method returns its result and ignores the DataLayout.
std::optional<int64_t> getSpecifiedBitwidth(const DataLayout& dl);
5

Why a ScalarTypeInterface is the most relevant aspect here?
Fundamentally, what do we want about element types of a vector?
Taking it from the other way around: if we’re not requiring a compile-time known size, then what is a type that shouldn’t be part of a vector for conceptual or practical reasons?

6

index is already allowed in vectors, and it has a size that will be known eventually before egress from MLIR.

The properties we want, I claim are

  1. At some point before egress, a bitwidth for this type will be chosen and stay fixed
  2. Said type has scalar/plain-old-data semantics - it’s a concrete things you can load/store to memory and could bitcast into an integer if you wanted to hack on the bits (ex. it’s not something like !async.token), and they’re not complex structures that might have padding etc. (so memref is out)

If we had a time machine, for example, and could do IntegerType with our modern type generation support, I wouldn’t have had uiN and siN, I’d have !c.signed<N> and !c.unsigned<N>, both implementing ScalarTypeInterface and having a known bitwidth.

Similarly, pointers are ScalarTypeInterface, even though their width is target-dependent. A pointer can be 32 or 64 bits, and at some point we’ll know which … but vector<4xptr> is still a meaningful concept before we’ve picked the width.

To answer the “what shouldn’t go in a vector” … at the very least, you can’t be both ShapedTypeInterface and ScalarTypeInterface for conceptual reasons

7

To give another concrete example of something that is ScalarTypeInterface, custom quantized or small float types. Both in the sense of wrappers around integers/floats that carry quantization information - the quantized types upstream are scalar. Similarly, if someone just invents a float datatype … fE7M16 or something, say, they can still have these arbitrary 24-bit values you can load, store, put in vectors, and pass to some custom operation without needing to make them official floats (both because that requires extending APFloat and because it implies ordinary arithmetic is meaningfully implemented on these things)

8

Why is that? A complex structure with padding seems to fit the definition of a POD that you can load/store to memory and cast to an integer (a large one but still).

That’s not really answering why though: why couldn’t I have a vector inside a vector for example? A ShapeTypeInterface can also be a “POD that you can load/store to memory and cast to an integer”.

Well I’m still more looking about the opposite: reasons to limit ourselves to “scalar” and why something should not be allowed in vectors.

9

I think that’s a very fair question, and my initial thought is this:

Property of scalars 3: they lower to something [at least one] egress dialect would consider a valid vector element

So index, pointers, and !quant<i8, scale=5, ...> - which’d lower to i8 (or, if it weren’t yet a built-in type, !fp8.f8E5M2 lowering to i8) are scalar by way of being meaningful vector elements somewhere, memref<> and vector<> themselves aren’t

2 Likes
10

Would it be desirable to deprecate getIntOrFloatBitWidth in favor of getInherentBitWidth?

There is llvm::TypeSize getTypeSizeInBits(Type t) const; is class DataLayout. Presumably, this interface method would just call getSpecifiedBitwidth?

11

I’m having a hard time finding the discussion where we allowed index in vectors. My vague recollection that it was indeed related to the availability of the data layout so we can compute the size.

As for the data layout itself being mandatory, I don’t think it is a good idea for the full generality of MLIR. Do we really care about index size for a TOSA to StableHLO converter? For CIRCT? There’s also a question of what goes into the data layout as the structure is flexible. It can well be present and not have the required information. On the other hand, I was entertaining the idea of having some sort of pass prerequisites: a pass such as lowering towards LLVM or anything size-aware may be able to check and immediately fail the pass if the required data layout entries are missing or otherwise unsupported.

Ultimately, I think we do want a VectorElementTypeInterface that makes vectors extensible similar to what we have for memrefs. Using it with custom FP formats downstream is a strong enough argument IMO. However, I also expect this interface to essentially be a method-less tag, a contract between the element type and the vector type. We need to spell out that contract clearly in the documentation and it has proved difficult so far.

This goes into the very old unresolved discussion of what is the fundamental difference between vector and tensor. Historically, the vague answers revolved around the fact that vectors are somehow connected to hardware vector registers whereas tensors were not. This supports having vectors of weird FP types, potentially index, but not vector-of-vector of vector-of-memref. This would ironically preclude something like vector<5xi3> since I’m not aware of any hardware that would support such things or even vector<4x4x4xf32> and the entire n-D vector idea. So it is some sort of generalization of what can be supported by a union of target hardware. I would suggest looking into and describing what kind of generalizations (larger sizes, ranks, bitwidth rounding) we accept or not if we want to keep the definition somehow connected to hardware.

The elemental type “having a size” formulation is basically derived from this connection. I do not believe it’s the right abstraction for vector element types though, or at least not a sufficient one. It’s roughly what we use for memref element types, in addition with the idea of having an in-memory representation.

At the same time, I don’t want this proposal to be blocked on the fundamental discussion that has been running for years as much as I’d love that discussion to be resolved. There are concrete needs, in particular, I believe not having a vector of pointers is one of the main reasons Triton is (ab)using tensor types to represent vectors. So I’d like to focus on two questions:

  1. If we want vector<ptr>, do we also want vector<memref>? Conceptually, both are “pointers”, but the latter will make it obvious that we instantly get vector<memref<vector<memref<...>>>> kind of nesting. If not, this may start laying ground for what not to accept as vector element.
  2. Given the original relation to hardware, can we get away with vector<index> and arithmetic on that before extracting elements and reinterpreting them as pointers? Ultimately, pointers are just integers…

Based on the consensus on the above questions, I think we should be able to make the decision while hopefully making progress on the harder question.

12

We still encoded something about this difference here: FAQ - MLIR

2 Likes
13

Here’s a PR that adds a ScalarTypeInterface: [mlir][IR] Add `ScalarTypeInterface` and use as `VectorType` element type by matthias-springer · Pull Request #132400 · llvm/llvm-project · GitHub

Is this description accurate? I tried to summarize what was written in this thread.

    Indication that this type is a scalar type.

    The bitwidth of a scalar type is a fixed constant but may be unknown in the
    absence of data layout information.

    Scalar types are POD (plain-old-data) entities that have an in-memory
    representation: scalar values can be loaded/store from/to memory, so
    abstract types like function types or async tokens cannot be scalar types.

EDIT: I removed this sentence because it is vague and does not give a clear answer:

    Scalar types should be limited to types that can lower to something that
    egress dialects would consider a valid vector element type.
14

Thanks for pushing on this, Matthias! Apologies for the delay, I was traveling and couldn’t reply earlier.

To bring more context, here are a few points we discussed during the Tensor Compiler WG on Wednesday:

  1. The RFC should focus on supporting vectors of pointers. The discussion about what THE pointer type in MLIR deserves its own discussion and RFC. From the vector type perspective, the actual pointer type chosen wouldn’t make a significant difference. Once the community decides on the pointer type for broader MLIR use and adoption, it should be straightforward to adapt the vector type accordingly.

  2. The same applies to opening the vector element type to an interface. That involves several challenges (e.g., vectors of aggregate types as Alex mentioned) which also deserve a separate discussion and RFC. Note that the need for an interface or trait to support vector of pointers arises to avoid dependencies between the LLVM/Pointer dialects and the Vector dialect, rather than the need for a vector element type generalization.

  3. As a practical first step, vector<llvm.ptr> seems like a reasonable choice since LLVM is the only(?) widely-adopted dialect with pointer support upstream. This will help resolve the issues on having vector and llvm.vector mentioned earlier and move the LLVM dialect to a better state. This is my concern right now. Once the ptr.ptr type is broadly adopted in MLIR, transitioning vector<llvm.ptr> to vector<ptr.ptr> (or anything else, really) should be straightforward.

Hopefully, this makes sense to people and we can keep the discussion a bit more focused. We can proceed incrementally with supporting more generic cases in the vector type.

I think using MLIR Core types in the LLVM dialect has proven to be effective in reducing unnecessary abstraction layers and exposing LLVM intrinsics earlier in the pipeline. I think going back to !llvm.vec would go in the opposite direction.

We considered something like this internally but it looks like a workaround to me which I think it’s what this RFC is trying to fix. If we use vector<index>, we would have to propagate the address space as well somehow…

1 Like
15

Agree. We made this mistake in llvm and it took years to fix. I’d rather not do it again in mlir.

16

Can you folks elaborate a bit here so, ideally with some references that we can put into the rationale doc when we make the decision?

2 Likes
17

Pointers are not just integers in the real world. Most common hardware is like that, but there is hardware where that is not true, whether it be CHERI, GPUs or something else.

1 Like
18

I think for this proposal it does not matter if pointers are integers or something more complex. This proposal does not assume that pointers are integers.

Is there any difference between VectorElementTypeInterface and ScalarTypeInterface that we discussed above?

How should we proceed with the RFC?

  1. Go with the original prototype and hard-code !llvm.ptr / PointerLikeTypeTrait as another allowed element type. Discuss ScalarTypeInterface in a separate RFC (or a continuation of this RFC).
  2. Finish the discussion around ScalarTypeInterface first, and then use that as vector element type. For this, it looks like we “just” have to agree what a scalar type is. Does somebody disagree with the above definition? Are there any other open questions that I missed?

Option 1 is a smaller, more incremental step and would be my preferred option. (But maybe there’s already consensus on Option 2 and we can go with that right away…)

19

This was in response to a suggestion to use index types as pointers. Not to your original proposal.

Unknown element types should be fine as long as we have a guarantee they’re contiguous in memory and have a known alignment.

You may dispatch to some runtime functions, but that must be known at compile time, or we have data layout, and can calculate.

20

My reading is that we are nearing some consensus on the definition of ScalarTypeInterface here, but I won’t oppose the pointer-specific change as long as we figure out the answer to the memref conundrum:

I have some nitpicks about the wording, specifically:

  • I’d avoid the notion of POD: we don’t want to conceptually depend on C++ semantics, and the notion itself is deprecated. Note that a memref descriptor, for example, is POD.
  • Renato’s wording looks reasonable to me, though it’s narrower than POD if I understand “contiguous” correctly.