Vector ABI mismatch during inlining

Hi,

Consider the following code:

target triple = “x86_64-unknown-linux-gnu”

define void @test1() “target-features”=“+avx” {
call void @test2()
ret void
}

define void @test2() {
call void @test3(<4 x i64> <i64 0, i64 1, i64 2, i64 3>)
ret void
}

define void @test3(<4 x i64> %arg) noinline {
ret void
}

Inlining will inline test2 into test1, because test1 has a superset of target features:

target triple = “x86_64-unknown-linux-gnu”

define void @test1() “target-features”=“+avx” {
call void @test3(<4 x i64> <i64 0, i64 1, i64 2, i64 3>)
ret void
}

define void @test2() {
call void @test3(<4 x i64> <i64 0, i64 1, i64 2, i64 3>)
ret void
}

define void @test3(<4 x i64> %arg) noinline {
ret void
}

Now we have a problem: X86 uses different vector ABIs depending on target features. test3 is compiled without avx, and as such expects the argument to be passed in two XMM registers. test2 is also compiled without avx and performs the call correctly. test1 on the other hand is compiled with avx and will instead pass a single YMM register.

Note that the by-value vector arguments do not necessarily have to be present in the original code – even if the frontend passes all vectors by memory to avoid precisely this issue, argument promotion can convert them to by-value arguments, as the caller/callee ABIs match at the time argument promotion runs (between test2 and test3).

I would like to have some input on how this miscompile could be addressed. The two general options I see are:

  1. Fix call lowering to respect callee target features. That is, even if the caller is +avx, if the callee is not then we should pass arguments via xmm rather than ymm. I’m not sure to what degree this is possible without our current infrastructure though.

  2. Prevent inlining in this case. I don’t think we can’t prevent inlining across functions with different target features entirely, as that would be a performance disaster. But possibly we can inspect the body of the callee to check for calls that would be problematic under the new ABI.

Regards,

Nikita

+echristo who implemented some of this stuff a while back - I don’t recall discussing this particular problem before, though…

(1) seems sort of plausible to me (speaking fairly naively) - do we correctly annotate function declarations with the attributes today? (eg: if test3 were defined in some other translation unit - would the declaration of test3 in this translation unit have the right avx attributes on it?)