While working on vector intrinsics, I found myself thinking about
splitting the Legalize phase of codegen into two separate phases,
legalizing types, and then legalizing operations in a separate
traversal. Has anyone else thought about doing this, or knows of
a reason why this would or wouldn't be beneficial?
Yes, that is an excellent idea.
One significant advantage of this is that we can then run a dag combine pass between the two. Right now, dag combine sometimes misses some xforms because the operation-lowering piece custom lowers some operations to target-specific dag nodes that dag combine doesn't understand. Splitting this up would allow an intermediate step for dag combine to simplify code.
Another thing on my long term todo list has been to switch legalize from being recursive to being iterative. Right now, on some very large basic blocks, legalize runs out of stack space (also, if you're running the JIT in a threaded program, the JIT often runs on thread stacks that are comparatively small, which greatly exacerbates the issue). Doing the first change would make the second one easier to do.