I've got it working on x86. Most of the logic is in SelectionDAGBuild
and friends, but there is one target-specific hook
(TargetLowering::CanLowerReturn) that needs to be implemented for each
target. It should return true if the function can return the value in
registers without crashing, and false if the function needs to have a
hidden sret-parameter inserted to do the return; i.e., if the
aggregate is too large to fit in the registers.
The implementation for x86 is very simple, and the implementation for
the other targets is *probably* pretty much the same. But I have no
way to test it, and no desire to break something. (If CanLowerReturn
returns false when it should return true, unexpected ABI mismatches
with native external functions will result.). So I will leave it to
those who have access to non-X86 machines to fill in the
CanLowerReturn implementation for those targets and provide test
cases. Once that is done, we can safely return large aggregates in IR
without crashing the code generator.