There were concerns about dialects having operations that cannot be lowered further down in the original discussion on splitting std - [RFC] Splitting the Standard dialect - #11 by rengolin. Having tensor_load and bufferize_cast in the hypothetical memref triggers those concerns.
I suppose it’s not necessarily a problem the conversion setup remains easy, e.g., all memref operations can be declared invalid in the conversion target either because they can be converted or because they should have been removed before the conversion starts.