Proposal
Hi all,
We would like to discuss upstreaming support for re-vectorisation in the LoopVectorizer. AArch64 NEON-to-SVE is the initial motivating target, but this could also apply to other targets that have legacy vector ISAs.
The short version is that many loops in real code already operate on fixed-length vectors, often because they were written using target intrinsics. Today the LoopVectorizer ignores such loops as already-vectorised and does not try to widen them further.
For Arm, this misses an important migration path: there is a lot of existing code written for NEON, and being able to turn fixed-length NEON-style vector loops into scalable-length SVE loops is an investment in future-proofing that code without requiring source rewrites.
The prototype in our downstream branch [1] adds this support under an opt-in flag: -vectorize-vector-loops. The normal LoopVectorizer behaviour remains unchanged unless this new mode is explicitly enabled. The intent is similar to SLPVectorizer’s -slp-revec which offers fixed-length to fixed-length re-vectorisation while fixed-length to scalable-length re-vectorisation is the primary goal here.
What we mean by re-vectorisation
As stated above, the goal is to widen fixed-length vectors to wider vectors. For the purpose of our use-case and the remainder of this proposal, the wider vectors are scalable vectors, although the concepts equally apply to wider fixed-length vectors. The way it works is to create a “hybrid” between vector-length-agnostic and vector-length-specific code: a fixed-length vector from the original loop becomes a “segment” inside a wider vector.
The LoopVectorizer can then widen the loop across multiple such segments using a scalable VF. For example, a “scalar” iteration that loads <8 x i16> may become a scalable operation over <vscale x <8 x i16>>, where each SVE vector contains one or more fixed-length NEON-sized segments. In many cases, operations on e.g. <vscale x <8 x i16>> are equivalent to operations on <vscale x 8 x i16>, but this isn’t always the case as will be explained in more detail later.
For an input “NEON” loop like:
for.body:
%iv = phi i64 [ 0, %entry ], [ %iv.next, %for.body ]
%p = getelementptr inbounds <8 x i16>, ptr %in, i64 %iv
%v = load <8 x i16>, ptr %p, align 16
%r = call <8 x i16> @llvm.aarch64.neon.uabd.v8i16(
<8 x i16> %v, <8 x i16> splat (i16 1))
%q = getelementptr inbounds <8 x i16>, ptr %out, i64 %iv
store <8 x i16> %r, ptr %q, align 16
%iv.next = add nuw nsw i64 %iv, 1
...
the re-vectorised loop for VF = vscale x 1 uses scalable SVE operations:
%wide.load = load <vscale x 8 x i16>, ptr %wide.ptr, align 16
%wide.op = call <vscale x 8 x i16> @llvm.aarch64.sve.uabd.nxv8i16(
<vscale x 8 x i1> splat (i1 true),
<vscale x 8 x i16> %wide.load,
<vscale x 8 x i16> splat (i16 1))
store <vscale x 8 x i16> %wide.op, ptr %wide.out, align 16
Why this belongs in the LoopVectorizer
The re-vectorisation transformation is still loop vectorisation: it needs the same legality checks, trip-count handling, runtime checks, VPlan construction, costing, and target queries as ordinary scalar-to-vector LV. The main difference is that the “scalar” value for the purpose of widening may itself be a fixed-length vector.
The prototype builds on the LoopVectorizer without requiring any major redesign. Most changes are generalisations of existing code paths from “element type” to “initial value type”, plus target hooks where legality or cost is target-specific:
- Legality detects fixed-length vector instructions only when
-vectorize-vector-loopsis enabled. Scalable vector inputs are still rejected so we do not try to re-vectorise code that is already scalable. - Most ordinary arithmetic, casts, selects, loads, stores, reductions, and first-order recurrences are handled by reusing the existing VPlan recipes after teaching type inference and widening helpers that the per-lane value may be a vector.
The places that need explicit re-vectorisation handling are mostly the places where LLVM IR currently has no way to express an operation over fixed-length vector segments. For example:
- Non-contiguous vector loads/stores use the same gather/scatter shape as ordinary non-contiguous scalar memory, but lower through segment-aware intrinsics because the data “lane” is a fixed vector segment.
- Interleaved groups reuse the existing interleaved-access representation, but use segment-level interleave/deinterleave intrinsics when the member type is a vector.
shufflevectorneeded a dedicated VPlan recipe because its semantics are about positions within the fixed-length vector segment.
The current prototype deliberately keeps unsupported cases conservative. For example, some segment operations are restricted to VF = vscale x 1 until there is agreement on the IR and lowering story.
Segments and endianness
It can be tempting to model <vscale x 4 x i32> as <vscale x 1 x i128> and reuse ordinary vector operations over a wider element type, as if the <4 x i32> segment were just a bag of bits. That is not always semantically equivalent because the original value is still four i32 lanes in vector lane order, while the container representation is one endian-sensitive integer element.
This matters most for operations that observe memory layout or lane order. A segment gather/scatter transfers one fixed-length vector segment for each active scalable predicate lane. A gather/scatter of a wider integer element would instead describe a different element type, and bitcasting around it does not preserve the same semantics on big-endian targets.
For example, the segment operation we want is:
; One predicate bit controls one address, and each active address transfers
; one fixed-length segment in vector lane order.
%seg = call <vscale x 4 x i32> @llvm.masked.segment.gather.nxv4i32.nxv1p0(
<vscale x 1 x ptr> align 16 %ptrs,
<vscale x 1 x i1> %mask,
<vscale x 4 x i32> %passthru)
The tempting container form is not the same contract:
; This describes one i128 element per active address, then reinterprets the
; result as four i32 lanes. On big-endian targets that container value does
; not have the same lane-order semantics as a segment operation.
%tmp = call <vscale x 1 x i128> @llvm.masked.gather.nxv1i128.nxv1p0(
<vscale x 1 x ptr> align 16 %ptrs,
<vscale x 1 x i1> %mask,
<vscale x 1 x i128> %passthru.i128)
%seg = bitcast <vscale x 1 x i128> %tmp to <vscale x 4 x i32>
Implementation details
This section gives technical details based on the prototype branch. The aim is to help readers understand the typical problems that come up when re-vectorising loops that already operate on fixed-length vectors, and to give a high-level overview of the changes made in the prototype.
We’d prefer to avoid discussing the details of the interfaces or intrinsics as shown in this RFC, as those can be discussed individually at a later point. Our primary aim is to get some initial feedback on the concept as a whole.
Click to show implementation details
TargetTransformInfo hooks
The prototype adds several TTI hooks. The default implementation either returns false or an invalid cost, so targets opt in explicitly.
Legality hooks
isLegalMaskedSegmentGather(Type *DataType, Align Alignment)isLegalMaskedSegmentScatter(Type *DataType, Align Alignment)
These answer whether a target can support masked gather/scatter where each active scalable lane transfers one fixed-length segment.
The AArch64 hooks allow segment memory operations only for little-endian targets. Segment gather/scatter and segment interleaved accesses map directly to quad-element or doubleword-element operations in ISel. That direct mapping is only appropriate for little-endian AArch64.
Costing hooks
getMemIntrinsicInstrCost(...)extended to support.segmentgather/scatter intrinsicsgetSegmentInterleavedMemoryOpCost(...)getSegmentedShuffleCost(...)
These separate segment operations from ordinary element operations. Treating a <vscale x <N x Ty>> operation as a normal <vscale x N x Ty> vector of elements would not necessarily yield the right costs.
Target-intrinsic hooks
isTargetIntrinsicVectorizable(Intrinsic::ID ID)vectorizeTargetIntrinsic(...)isSupportedTargetRecurrence(Intrinsic::ID ID, RecurKind RK)
The first two let the target declare that a target-specific fixed-length intrinsic can be widened and then materialise the widened target-specific operation. In the AArch64 prototype this is used to map llvm.aarch64.neon.* intrinsics to llvm.aarch64.sve.* intrinsics. Our goal is to allow both the LoopVectorizer and the SLPVectorizer to rely on these hooks. Currently, there is poor support for preserving flags and metadata, but we’ll make sure to account for that when productising.
isSupportedTargetRecurrence is used for target-specific reductions such as NEON dot-product accumulations (udot, sdot, usdot) where the first operand is the accumulator.
Segment intrinsics
Several generic intrinsics were introduced in the prototype to represent operations over fixed-length segments inside scalable vectors.
Masked segment gather/scatter
declare <vscale x 8 x i16> @llvm.masked.segment.gather.nxv8i16.nxv1p0(
<vscale x 1 x ptr> align 16 %ptrs,
<vscale x 1 x i1> %mask,
<vscale x 8 x i16> %passthru)
declare void @llvm.masked.segment.scatter.nxv8i16.nxv1p0(
<vscale x 8 x i16> %data,
<vscale x 1 x ptr> align 16 %ptrs,
<vscale x 1 x i1> %mask)
The important semantic point is that the predicate is <vscale x 1 x i1>: each predicate lane guards one fixed-length segment, not one scalar element.
We may also want masked segment load/store forms, even for contiguous memory, to make the intended predicate type explicit and avoid relying on later combines.
Segment interleave/deinterleave
The prototype adds segment variants of the existing vector interleave intrinsics:
declare <vscale x 8 x i16> @llvm.vector.interleave.segments2.nxv8i16(
<vscale x 4 x i16> %a,
<vscale x 4 x i16> %b)
declare { <vscale x 4 x i16>, <vscale x 4 x i16> }
@llvm.vector.deinterleave.segments2.nxv8i16(<vscale x 8 x i16> %v)
These operate on fixed-length segments rather than scalar elements. They are used for interleaved memory groups whose members are fixed vectors, and for length-changing shufflevector idioms such as subvector concatenation and subvector extraction.
For example:
%concat = shufflevector <4 x i16> %a, <4 x i16> %b,
<8 x i32> <0, 1, 2, 3, 4, 5, 6, 7>
can become:
%wide.concat = call <vscale x 8 x i16>
@llvm.vector.interleave.segments2.nxv8i16(
<vscale x 4 x i16> %wide.a,
<vscale x 4 x i16> %wide.b)
and:
%extract = shufflevector <8 x i16> %v, <8 x i16> poison,
<4 x i32> <0, 1, 2, 3>
can become:
%parts = call { <vscale x 4 x i16>, <vscale x 4 x i16> }
@llvm.vector.deinterleave.segments2.nxv8i16(<vscale x 8 x i16> %wide.v)
%wide.extract = extractvalue { <vscale x 4 x i16>, <vscale x 4 x i16> } %parts, 0
The design question is whether these are sufficiently general to justify generic IR intrinsics. Their advantage is that they avoid introducing wider element types such as i128/i256.
Segmented shuffle
Length-preserving shufflevector instructions are represented with:
declare <vscale x 8 x i16> @llvm.vector.segmented.shuffle.nxv8i16.v8i32(
<vscale x 8 x i16> %src0,
<vscale x 8 x i16> %src1,
<8 x i32> %mask)
The fixed-length mask is applied independently to each segment. This is useful for NEON idioms such as unzip, transpose, reverse within a 128-bit segment, duplicate-element shuffles, and other lane rearrangements that should not cross segment boundaries.
For example:
%r = shufflevector <8 x i16> %v, <8 x i16> poison,
<8 x i32> <0, 2, 4, 6, 0, 2, 4, 6>
becomes:
%wide.r = call <vscale x 8 x i16>
@llvm.vector.segmented.shuffle.nxv8i16.v8i32(
<vscale x 8 x i16> %wide.v,
<vscale x 8 x i16> poison,
<8 x i32> <0, 2, 4, 6, 0, 2, 4, 6>)
Vector stretch
The prototype uses llvm.vector.stretch to duplicate each input element in place until the result type is reached:
%wide.mask = call <vscale x 8 x i1>
@llvm.vector.stretch.nxv8i1.nxv1i1(<vscale x 1 x i1> %segment.mask)
This is mainly useful when a segment-level predicate has to guard an element-level operation such as a select or a masked contiguous load/store. It also keeps VPlan emission simple by representing “broadcast this predicate over the fixed-length segment” directly in IR.
This is also the mechanism used by the prototype for some if-converted vector loops: a segment predicate can be stretched into an element predicate at the point where an element-wise operation needs it. Tail folding is currently disabled for re-vectorised loops, so the initial upstream behaviour would rely on the usual scalar epilogue path for tails. That is a conservative starting point rather than a fundamental limitation.
Note that the intrinsic is not strictly required because it is semantically equivalent to repeatedly interleaving a vector with itself. For example, stretching <vscale x 1 x i1> to <vscale x 4 x i1> could be written as a vector.interleave4 operation. The problem is that larger fixed segments may need higher factors such as interleave8 or interleave16, at which point the IR gets difficult to read.
Vector broadcast
The prototype introduces llvm.vector.broadcast to broadcast a fixed-length vector value into a wider scalable vector of segments:
%wide.v = call <vscale x 8 x i16>
@llvm.vector.broadcast.nxv8i16.v8i16(<8 x i16> %v)
This comes up for uniform or live-in fixed-length values that need to be broadcast VF times. It is conceptually the vector-valued analogue of a scalar splat.
Partial reductions
Re-vectorising a fixed-vector reduction must still produce a fixed vector at loop exit. The prototype currently relies on partial reduction intrinsics to reduce the widened vectors back to their original types.
For example, if the original loop accumulates <4 x i32> values, the vector loop carries a <vscale x 4 x i32> accumulator. In the middle block, the scalable accumulator is reduced across the scalable VF back to one <4 x i32> segment, and that segment is then combined with the scalar-loop start value:
vector.body:
%vec.phi = phi <vscale x 4 x i32> [ %start.splat, %vector.ph ],
[ %vec.next, %vector.body ]
%wide.load = load <vscale x 4 x i32>, ptr %p
%vec.next = add <vscale x 4 x i32> %vec.phi, %wide.load
...
middle.block:
; Reduce across the scalable VF, preserving the original <4 x i32> segment.
%segment.sum = call <4 x i32> @llvm.vector.partial.reduce.add.v4i32.nxv4i32(
<4 x i32> zeroinitializer,
<vscale x 4 x i32> %vec.next)
However, we do not want to change the existing semantics for partial.reduce intrinsics and require them to preserve lane-ordering within segments. Therefore, we’ll need a different implementation when productising, e.g. relying on new intrinsics or a new TTI hook to query the target directly about how to reduce values.
AArch64 target intrinsics
The AArch64 implementation uses a TableGen mapping from NEON intrinsics to SVE intrinsics where possible. The table records the target intrinsic name and how to construct the SVE argument list from the widened NEON arguments. The mapping language supports:
A:N: use widened argumentN.T: synthesise an all-true predicate of the appropriate SVE predicate type.Z: synthesise a zero value.P: synthesise poison.I:N: scalarise a constant splat into an immediate operand.
A mapping entry is intentionally small. For example:
// uabd(a, b) becomes sve.uabd(ptrue, a, b).
def : IntrinsicMapping<"llvm.aarch64.neon.uabd",
"llvm.aarch64.sve.uabd",
"T,A:0,A:1">;
This covers direct mappings such as dot products, absolute difference, halving/rounding add/sub, saturating add/sub, saturating shifts, widening/narrowing operations, MMLA operations, and many conversion intrinsics.
Some NEON intrinsics are marked as custom because the corresponding SVE sequence needs extra IR.
Current limitations
The prototype is intentionally conservative. Known restrictions include:
- It only considers loops containing fixed-length vector values; loops already containing scalable vectors are rejected.
- Tail folding is disabled for re-vectorised loops.
- Some segment interleave/deinterleave cases are restricted to
VF = vscale x 1. - Reverse interleaved segment groups are not supported yet.
- Support for length-changing
shufflevectorpatterns is limited to “aligned” segment concatenations or extracts.
Upstream integration plan
The upstream work would naturally split into a few pieces, some of which could be worked on in parallel:
- An opt-in LoopVectorizer mode, exposed through
-vectorize-vector-loops, with the legality and type-inference changes needed to recognise fixed-length vector loops as re-vectorisation candidates. - New IR intrinsics for segment operations that cannot be expressed cleanly with existing vector intrinsics, including LangRef/verifier coverage and target-independent lowering support where needed.
- TTI hooks with conservative default behaviour, so targets explicitly opt in to the legality and costing decisions that depend on their scalable vector ISA.
- Target-independent VPlan support for widening fixed-vector values, including segment gathers/scatters, segment interleaves, reductions, and
shufflevector. - The remapping of AArch64 NEON intrinsics to SVE equivalents.
Everything should remain disabled by default under -vectorize-vector-loops for the initial upstream work. The first upstream target would be AArch64, but the generic pieces are intended to be target-independent. Other targets could opt in by implementing the TTI hooks and lowering the segment intrinsics that make sense for their vector ISA.
Questions for the community
As a first step, we’d like to get some input on whether we can pursue this direction upstream, and specifically hear if there are any objections. Is re-vectorising fixed-length vector loops in the LV initially behind -vectorize-vector-loops a reasonable direction?
In the coming weeks, we’re hoping to get some feedback here in the RFC or via the upcoming Vectorizer community call, and then we’d like to start the conversation about some of the technical design considerations for the implementation.
Cc @fhahn @david-arm
The prototype branch is at [DRAFT] Re-vectorisation in LV by gbossu · Pull Request #203885 · llvm/llvm-project · GitHub
I agree we should not stretch that contract too much and probably move the codegen hook elsewhere. I’m sure we’ll have time to discuss those details in PRs.