function returning small-ish struct

In the case of a struct small enough to fit in the arg-registers, I want my calling-convention to do function-result struct like function-arg struct, pass the result back in the arg-registers, and only resort to pass-by-reference in the case of large structs.

Has anyone else done this in an llvm backend, if so are the relevant source files in the public domain, and can you point me to them.

Also if so, is your solution entirely in the backend, or do you modify clang as well (to not use “hidden pointer argument”).

Thanks,

Peter Lawrence.

Has anyone else done this in an llvm backend, if so are the relevant source
files in the public domain, and can you point me to them.

This is pretty common. Both ARM targets certainly do it. For example
in lib/Target/AArch64/AArch64CallingConvention.td, RetCC_AArch64_AAPCS
returns i64 arguments in X0-X7 even though real C integers arguments
could only ever use X0 & X1 (and that's for __int128). The rest come
into play for larger structs.

Also if so, is your solution entirely in the backend, or do you modify clang
as well (to not use “hidden pointer argument”).

You definitely don't want Clang using a hidden pointer argument for
this; it would be really tricky to undo that in the backend. Normally
the Clang-side involves picking a "close enough" scalar type so that
the usual backend rules kick in properly. (things like iN, [N x iM]
are useful here).

Cheers.

Tim.