We have such a dialect: the vector dialect (“vectors” is a historical misnomer here, since they are really n-dimensional constructs; but they differ from the tensor type, so the name was already taken). Have you read @nicolasvasilache’s brilliant Vector design doc, the earlier AVX512 case studies, or any of the RFCs for Arm Neon/SVE dialects, and AVX512 dialect?
In a nutshell, the architectural-agnostic vector dialect (VV for Virtual Vector level) is used as long as possible during rewriting and progressive lowering. In the end, an architectural-specific vector dialect (HWV for Hardware Vector level) may (or may not) be used to take advantage of particular idioms. For many case, we found that extremely good performance can indeed be obtained with just VV, and relying on the LLVM “backend” to generate efficient code for the SIMD IR that is passed on. For very hardware specific features (AMX being a very clear example I would say), we introduce an HWV, at least until such a time the lowering to LLVM is better understood.