X86_64 ABI - Non-determiistic coersion

In the case of the following enum, result of coersion depends on the order of union elements, which IMHO, is counter intuitive.

typedef union {
void* INT[2];
long double X87_X87UP;
double SSE;
} InReg;

InReg in_reg(InReg x) { return x; }

typedef union {
long double X87_X87UP;
double SSE;
void* INT[2];
} InMem;

InMem in_mem(InMem x) { return x; }

In IR these functions have the following signatures and are binary incompatible:

define { i64, i64 } @in_reg(i64 %x.coerce0, i64 %x.coerce1)
define void @in_mem(%union.InMem* noalias sret %agg.result, %union.InMem* byval align 16 %x)

This does not conflict with ABI specification, because the later does not specify order in which union elements should be visited.

Do you think this is a real problem that is worth fixing in the ABI?

It is possible to fix this in the ABI by:
a) formalizing order in which fields are visited
b) changing merging and post-processing rules to have a consistent result

The first option is backward-compatible, the second allows to achieve better performance of argument passing (I’m not expert in this field, but I assume there are some performance concerns behind the current argument classification rules).

Below is proposed algorithm to perform merging with consistent result:

To end up with { MEMORY, MEMORY } in the end:

  • Add two more classes NO_SSE and NO_X87

  • Merging INT with X87, X87_UP, COMPLEX_X87 results in NO_SSE

  • Merging INT with SSE or SSEUP results in NO_X87

  • Merging NO_SSE with SSE results in MEMORY

  • Merging NO_X87 with X87, X87_UP or COMPLEX_X87 results in MEMORY

  • Merging NO_SSE with NO_X87 cannot occur, because these classes may appear only as a result of merging - no field may have a class of NO_???

  • During post-merge phase NO_SSE and NO_X87 are replaced with INT

To end up with { INT, INT } in the end:

  • Add one more class MAYBE_MEMORY
  • Merging X87/X87_UP/COMPLEX_X87 with SSE/SSEUP results in MAYBE_MEMORY
  • Merging MAYBE_MEMORY with INT results in INT
  • During post-merge MAYBE_MEMORY is replaced with MEMORY