[LoadStoreVectorizer] Requesting review: pointer recovery in mixed-type vectorization (inttoptr/ptrtoint)

Hi,

I’m working on PR #177908, which extends the LoadStoreVectorizer (LSV) to merge contiguous loads/stores of different scalar types (e.g. {ptr, i16, i16} or {ptr, <32 x i1>}) into a single vector memory operation. This resolves #97715.

Most of the PR review feedback has been addressed, but one item remains open: whether the pointer-recovery strategy in mixed-type chains is IR-legal and provenance-safe. I’m posting here to get broader input before merging.


Background

LSV currently groups instructions by base pointer and address space. This PR removes scalar element type from the equivalence-class key so mixed-type contiguous accesses can be vectorized together.

For a heterogeneous chain, the vector lane width is the GCD of the scalar bitwidths. Original types are recovered after the vector load/store using IRBuilder::CreateBitPreservingCastChain.

Homogeneous chains (e.g. {i32, i32, i32, i32}) keep their original scalar element type to avoid unnecessary churn.


The pointer concern

Mixing pointers with narrower types is the hard case. An early approach normalized chains to integer GCD lanes (iN / <K x iN>) and used inttoptr/ptrtoint to recover pointers. That is incorrect when the pointer is wider than the GCD:

; GCD = 32, ptr = 64-bit — loses high bits
%trunc = trunc i64 %p to i32
%ptr   = inttoptr i32 %trunc to ptr   ; unsound

To avoid that, the PR now always normalizes heterogeneous chains to byte lanes (b<GCD> / <K x b<GCD>>), including sub-byte widths (b1, b2, b4). LangRef permits pointer↔byte bitcasts of the same size, and byte types represent raw memory bits without integer provenance.

Example from llvm/test/Transforms/LoadStoreVectorizer/mixed-ptr-and-subbyte-int.ll:

; Before
%x0 = load ptr addrspace(3), ptr addrspace(3) %o0
%x1 = load <32 x i1>, ptr addrspace(3) %o4

; After LSV
%v  = load <64 x b1>, ptr addrspace(3) %o0
%p0 = shufflevector <64 x b1> %v, ..., <32 x i32> <0, 1, ..., 31>
%p  = bitcast <32 x b1> %p0 to ptr addrspace(3)
%i1 = bitcast <32 x b1> %p1 to <32 x i1>

Similar coverage exists in mixed-ptr-and-subbyte.ll and AMDGPU/mixed-ptr-and-narrow-int.ll.


Where inttoptr/ptrtoint still appears

Reviewers flagged that CreateBitPreservingCastChain may still route some conversions through inttoptr/ptrtoint rather than a plain bitcast. Related work landed separately in #209557 (merged): byte and byte-vector types are now treated as “int-like” in CreateBitPreservingCastChain, so <N x b32> → ptr goes through bitcast to i64 + inttoptr instead of a direct (and invalid) bitcast <2 x b32> to ptr.

For the LSV sub-byte cases above, the tests currently expect direct same-sized bitcasts such as bitcast <32 x b1> to ptr. After rebasing fully onto main (with #209557), some of these may become inttoptr (bitcast … to iN) to ptr instead.

The question is whether that is acceptable in this context:

  1. Provenance: Does recovering a pointer from a byte vector (after a mixed-type vector load of raw memory) require preserving provenance via bitcast, or is inttoptr of the same bit pattern sound?
  2. LangRef: Is bitcast <N x bK> to ptr (same total bitwidth) the preferred form, or should we always go through inttoptr/ptrtoint for byte→pointer conversions?
  3. Sub-byte lanes: For GCD = 1/2/4, is normalizing to b1/b2/b4 (rather than i1/i2/i4) the right long-term approach?

What we’d like feedback on

  1. Is the byte-normalization strategy for heterogeneous LSV chains reasonable?
  2. For pointer recovery from byte vectors of the same total size, is bitcast vs inttoptr/ptrtoint the correct lowering?
  3. Any objections to merging #177908 once this is resolved?

Related PRs / context

NVIDIA review (@dakersnar) is tentatively positive pending this item; the original inttoptr concern on the sub-byte tests was resolved once we switched to byte normalization, but a second opinion from IR/provenance folks was requested.

cc @nikic @efriedma-quic @arsenm @dakersnar — any thoughts appreciated.

Thanks,
Anshil Gandhi

It depends a bit on exactly what you are doing, but generally it requires bitcast. Using inttoptr requires that the provenance was previously exposed via ptrtoint. If your overall transform ends up inserting both ptrtoint and inttoptr, then it’s correct, but sub-optimal (though you may not notice right now because we currently incorrect remove ptrtoint/inttoptr round-trips). If you don’t insert both, then it’s most likely entirely incorrect.

Doing the direct bitcast is the preferred form. (Though I think the backend currently crashes when you use it…)

Yes, at least if one of the original types is a pointer or byte type. Also if there are poison-propagation concerns, but I think that doesn’t apply in your case, as your element sizes can only be smaller. I think just always using the byte type if you have mixed types probably makes the most sense.

2 Likes

Just wanted to add that this idea of being able to vectorize pointer and int load/stores using the byte time is relevant for the SLPVectorizer as well. I recently merged an SROA improvement: https://github.com/llvm/llvm-project/pull/219411. The one downside is it led some allocas to be loaded with pointer type instead of int, leading to less vectorizations in some cases in SLPVectorizer.

1 Like

I see. So the following LoadStoreVectorizer transform looks fine to you, right?

define void @load_ptr32_v32i1(ptr addrspace(3) nocapture readonly align 8 dereferenceable(8) %blob) {
; CHECK-LABEL: define void @load_ptr32_v32i1(
; CHECK-SAME: ptr addrspace(3) readonly align 8 captures(none) dereferenceable(8) [[BLOB:%.*]]) {
; CHECK-NEXT:  [[ENTRY:.*:]]
; CHECK-NEXT:    [[O0:%.*]] = getelementptr inbounds i8, ptr addrspace(3) [[BLOB]], i64 0
; CHECK-NEXT:    [[TMP0:%.*]] = load <64 x i1>, ptr addrspace(3) [[O0]], align 8
; CHECK-NEXT:    [[X01:%.*]] = shufflevector <64 x i1> [[TMP0]], <64 x i1> poison, <32 x i32> <i32 0, i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7, i32 8, i32 9, i32 10, i32 11, i32 12, i32 13, i32 14, i32 15, i32 16, i32 17, i32 18, i32 19, i32 20, i32 21, i32 22, i32 23, i32 24, i32 25, i32 26, i32 27, i32 28, i32 29, i32 30, i32 31>
; CHECK-NEXT:    [[TMP1:%.*]] = bitcast <32 x i1> [[X01]] to i32
; CHECK-NEXT:    [[TMP2:%.*]] = inttoptr i32 [[TMP1]] to ptr addrspace(3)
; CHECK-NEXT:    [[X12:%.*]] = shufflevector <64 x i1> [[TMP0]], <64 x i1> poison, <32 x i32> <i32 32, i32 33, i32 34, i32 35, i32 36, i32 37, i32 38, i32 39, i32 40, i32 41, i32 42, i32 43, i32 44, i32 45, i32 46, i32 47, i32 48, i32 49, i32 50, i32 51, i32 52, i32 53, i32 54, i32 55, i32 56, i32 57, i32 58, i32 59, i32 60, i32 61, i32 62, i32 63>
; CHECK-NEXT:    ret void
;
entry:
  %o0 = getelementptr inbounds i8, ptr addrspace(3) %blob, i64 0
  %o4 = getelementptr inbounds i8, ptr addrspace(3) %blob, i64 4
  %x0 = load ptr addrspace(3), ptr addrspace(3) %o0, align 8
  %x1 = load <32 x i1>, ptr addrspace(3) %o4, align 4
  ret void
}

The first load element is extracted via a bitcast <32 x i1> to i32 and an inttoptr i32 to ptr addrspace(3).

No, this is incorrect. The bitcast + inttoptr should be just a bitcast.

1 Like

IRBuilder::CreateBitPreservingChain needs a fix then. I will create a PR to ensure it abides by the pointer provenance rules.

Of possible interest to you is my RFC (and implementation) for generalizing CreateBitPreservingChain into an actual full SROA replacement utility – which actually does SROA, and not our current pass named SROA which does a lot of random things semi-related to SROA – that aims to properly rewrite any type into any other type, while also being careful around the inttoptr rules (aka not adding them unless specifically required by the required input or output type – never just for implementation convenience) at every intermediate stage:

In particular for this use case, I’d envisioned that LLVM can be more reckless with using FCA (to preserve ptr provenience semantics) at all stages in the optimization pipeline, while expecting backends to be able to normalize it into a single register by using this kind of transform as part of isel and/or legalization. Depending on where it is used, the CreateBitPreservingChain helper can violate the rules around non-integral pointers, which forbids backends optimizations from introducing new inttoptr, ptrtoint or addrspacecast calls which weren’t originally present in the IR (it assumes the pass has already checked any necessary invariant like that before calling that builder) per the Unstable pointer representation section.

If this is of interest for collaboration, feel free to DM me to discuss progressing that.

(I also have an incidental bugfix for CreateBitPreservingCastChain in there for unusual float-to-ptr appearing in there, which would further change the check in #209557, and which I really should make a separate PR for someday – when I theoretically have lots of time for side projects.)

1 Like

Created a PR: [IRBuilder][GlobalISel] Add byte ptr cast support in IRBuilder by gandhi56 · Pull Request #222049 · llvm/llvm-project · GitHub. Any review comments will be highly appreciated.