Summary: some types need more contextual information to properly legalize than what TLI queries give.
Example: consider v32i1:
1. v32i1 = compare v32i32, v32i32 ; all good
2. v32i1 = compare v32i16, v32i16 ; problem: v32i16 not legal, but v32i1 is
The type v32i1 is legal because there it can be represented in a register and can be a result of a legal comparison of legally-typed vectors. But it can also be a result of comparing illegally-typed vectors, which causes a problem.
The type legalization first looks at the value type, sees v32i1, takes note that it's legal. Then, for the second case, it will attempt to legalize the operand types, but that action would also cause the result type to change (e.g. widen everything: v32i16->v64i16, v32i1->v64i1), hitting an assertion.
I'm seeing this issue on Hexagon, but it seems to be fairly generic in its nature. Has anyone else experienced this?
I want to fix this somehow. My current idea is to simply allow a known legal type to be replaced with another legal type, but there may be different approaches, like using SDValue in the TLI.whatAbouThisType family.
Does anybody have concerns, thoughts?