The @llvm.vector.splice intrinsic currently only accepts immediate offset arguments. There is a need in the Loop Vectorizer to perform a splice with a variable offset, so this RFC proposes to allow non-immediate offsets and in doing so split the intrinsic into two:
declare <4 x i32> @llvm.vector.splice(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg %imm)
->
declare <4 x i32> @llvm.vector.splice.down(<4 x i32> %vec1, <4 x i32> %vec2, i32 %offset)
declare <4 x i32> @llvm.vector.splice.up(<4 x i32> %vec1, <4 x i32> %vec2, i32 %offset)
Background
The @llvm.vector.splice intrinsic concatenates two vector arguments together, shifts them by an offset specified by %imm and then extracts either the lower or upper half of the vector.
It has two modes of operation depending on whether the offset is positive or negative. If positive, the elements are shifted downwards and the lower half is extracted. If negative, the elements are shifted upwards and the upper half is extracted.
Currently only RISC-V and AArch64 implement lowerings for this intrinsic. For other targets it is expanded via the stack.
The splice intrinsic is only used in one place within LLVM itself, in the Loop Vectorizer. It’s used to handle first-order recurrences for scalable vectors (where shufflevector isn’t available) to splice together a single element from the previous iteration with the remaining elements from another vector, and is always passed in an immediate argument of -1, e.g:
%x = call <vscale x 1 x i64> @llvm.vector.splice(<vscale x 1 x i64> %recur, <vscale x 1 x i64> %new, i32 -1)
But the splice intrinsic can also represent “sliding up” or “sliding down” a single vector when one of the operands is poison, and this maps particularly well to RISC-V’s vslideup.vi/vslidedown.vi instructions:
; %x = <a, b, c, d>
; A positive offset slides %x down
@llvm.vector.splice(%x, poison, 2) = <c, d, poison, poison>
; A negative offset slides %x up
@llvm.vector.splice(poison, %x, -2) = <poison, poison, a, b>
For RISC-V we would like to be able to use @llvm.vector.splice in the loop vectorizer to slide down a vector like so when EVL tail folding. Unfortunately the offset won’t be known at compile time and depends on the runtime value of @llvm.experimental.get.vector.length, so we need to relax the immediate constraint on the offset argument.
However just relaxing the immediate constraint quickly runs into the issue where @llvm.vector.splice is really two functions encoded into one. Backends generate different lowerings depending on the sign of the offset, so to handle variable arguments we need to generate both the “shift up” and “shift down” behaviours and add a runtime select between the two.
Proposed solution
This RFC proposes two sequential changes:
- Split @llvm.vector.splice into @llvm.vector.splice.up and @llvm.vector.splice.down
- Remove the immarg constraint on the offset
Splitting into down and up intrinsics
In the first change we would remove the old splice intrinsic and replace it with two new intrinsics:
declare <4 x i32> @llvm.vector.splice(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg %offset)
-->
declare <4 x i32> @llvm.vector.splice.down(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg %offset)
declare <4 x i32> @llvm.vector.splice(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg -%offset)
-->
declare <4 x i32> @llvm.vector.splice.up(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg %offset)
This would have no affect on codegen, but is necessary to be able to handle variable offsets without a runtime select in the next step.
The offsets would still be immediates but now they would always be treated as unsigned. And because the offsets are still immediates, AutoUpgrade can rewrite the intrinsic calls and declarations to their up/down equivalent depending if the offset was positive or negative.
One thing to note is that @llvm.vector.splice.down and @llvm.vector.splice.up are parallels to @llvm.fshl and @llvm.fshr, but operating on vector elements (similar to @llvm.cttz and @llvm.experimental.cttz.elts). Splitting them out into the two directions makes them more consistent in this regard.
Hardware instructions also tend to differentiate between the shift directions, e.g. vslideup on RISC-V, palignr on X86, and splice on AArch64 all shift upwards (or to the right).
Removing the immediate argument restriction
In a separate change we can remove the immediate offset restriction. Targets that don’t support variable offsets like AArch64 can reuse the generic lowering that goes through the stack. We would need to also update the cost model to reflect this.
declare <4 x i32> @llvm.vector.splice.down(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg %offset)
-->
declare <4 x i32> @llvm.vector.splice.down(<4 x i32> %vec1, <4 x i32> %vec2, i32 %offset)
declare <4 x i32> @llvm.vector.splice.up(<4 x i32> %vec1, <4 x i32> %vec2, i32 immarg %offset)
-->
declare <4 x i32> @llvm.vector.splice.up(<4 x i32> %vec1, <4 x i32> %vec2, i32 %offset)
The old intrinsic didn’t specify the behaviour when the offset was out of the range of [0, N) for a vector of <N x Ty>, which can happen at runtime with vscale, but with the new intrinsic can happen even for fixed-length vectors. We would propose to return poison for an out of bounds offset to match @llvm.vector.extract and friends.
After this the loop vectorizer will be able to use the intrinsic when tail folding on RISC-V, and it should hopefully provide another generically useful permutation primitive for scalable vectors. Most importantly existing downstream users of llvm.vector.splice shouldn’t be affected if the AutoUpgrade goes smoothly.
A rough draft of the two changes required is available to view here. @llvm.experimental.vp.splice should also be updated separately to match, but as of now there’s no direct need for an variable offset version.
Alternatives considered
Keep one intrinsic but add a i1 immarg is_upwards flag
This is an alternative to splitting out the intrinsic into two. It makes the AutoUpgrade process a bit easier, but we lose the symmetry with @llvm.fshl/@llvm.fshr.
Keep one intrinsic and emit a runtime select
Emitting a runtime select to choose between the positive and negative offset modes will incur a performance cost. It also seems very unlikely that any user will want this behaviour dynamically, as it is something defined in the LLVM LangRef and not a common hardware operation.
.