GPU PGO currently counts active lanes. I propose adding a separate count of how often a wave executes a measured block. Lane counts remain useful for branch probabilities, but wave counts can give spill placement a better estimate of how often a spill instruction executes.
Why lane counts are not enough
Suppose 100 waves of 32 lanes each take this branch. In every wave, 31 lanes take A and one takes B:
if (lane_id < 31)
A();
else
B();
C();
The lane counts are A = 3,100 and B = 100, but both blocks execute in all 100 waves. A lane-based frequency puts B at 100 / 3,200, or about 1/32 of entry. The proposed spill consumer uses its wave count instead: 100 / 100, or once per entry wave. A spill on B would be issued in every wave, not one in 32. Its memory traffic can still be lower because only one lane is active, so wave counts do not measure the entire spill cost.
A partial-wave loop shows why dividing lane counts by 32 is not a fix. If one active lane in each of 100 waves executes a loop body ten times, the body has 1,000 lane visits and 1,000 wave visits. Dividing the lane count by 32 would estimate only about 31 wave visits.
Wave visits cannot replace lane counts in the existing profile: the branch above has 100 wave visits, yet both successors have 100. Wave counts do not obey the scalar flow rules used for branch probabilities. Profile-derived uniformity also cannot supply this number; it classifies observed lane activity rather than counting wave visits.
Wave-profile metadata
The proposed IR format has two version-2 metadata nodes. Every field is an i64:
wave.profileon a function definition is[version, function-name hash, counts...]. The counts are indexed by original block identity; identity zero is the original entry count. If that count was not measured, the loader omits the profile.wave.profile.blockon a block’s terminator is[version, function-name hash, block identity, measured flag, successor identities...]. The successor order matches the terminator.
A wave count increases once each time a wave with at least one active lane reaches the measured point, including repeated loop visits. The measured flag distinguishes an observed count of zero from an unavailable count. Blocks without direct instrumentation, or whose identity or edges no longer match, have no usable wave count. Counts are hints for cost decisions, not proof that a path is uniform or that a transformation is legal. The original entry count stays the normalization point even if later passes remove that block.
Use and implementation
The producer adds a wave counter beside each existing GPU PGO counter; one active lane increments it once. The loader records direct counts and marks other blocks unmeasured. CFG passes keep counts only when a block still represents the same executions.
The opt-in AMDGPU SpillPlacement consumer (-mllvm -enable-wave-profiled-spill) scales valid block-wave counts against the original entry count. It requires a unique matching machine block with matching CFG edges. Where a count is missing or rejected, it uses the existing MBFI cost. General block frequencies and branch probabilities are unchanged. This first consumer has not yet established an isolated performance benefit.
The open PRs cover counters (#225589), metadata (#225837), loading (#225911), CFG preservation (#226185), and the draft spill consumer (#226215).
Questions
- Is a separate wave-visit count useful enough for spill costs, given that lane activity still affects memory traffic?
- Is the function count table plus per-block identity, measured flag, and successor list the right way to keep valid measurements and reject stale ones?