The ABI boundary lowering requires types to be broken down further into 'legal types' that can be mapped to registers. The secondary breakdown is currently handled by TargetLowering::LowerCallTo() calling getRegisterType() and getNumRegisters(). Most ABIs are defined in terms of C types, not LLVM IR types, so there is a close connection between the C frontend and the ABI lowering code in the instruction selector. It would be a good idea to have the ABI lowering code work independently of the type system used during instruction selection. This is made possible by having ABI lowering be part of the LLVM IR to MI translation process.
Between virtual registers being typed with EVTs instead of register classes and the greater flexibility in what is a "legal" type, the ABI lowering should become significantly easier for weird targets. Will LowerFormalArguments and friends still only see legal types, or will they see the raw LLVM IR or unlegalized EVT types?
I would like the ABI lowering to be isolated from the rest of the instruction selector as much as possible. I don’t think it should use the instruction selector’s notion of what is a legal type. I is not completely clear how this should work, we do need some way of breaking down weird types into chunks the target ABI code understands.
As I see it, types picked by the frontend to match ABI requirements should be passed directly to the target ABI code. Weird types created by inter-procedural optimizations are less critical to get right, but they must be lowered into something.
This is where the iterative legalization comes in. The legalizer would initially widen an i8 operation by using anyext/trunc operations which may not be legal. These operations may combine with loads and stores to form extload/truncstore, or they may be legalized some other way.
I imagine that anyext/trunc operations would also be pushed around the CFG, like the i64 type split in the second example in the proposal.