You could use the same trick as SmallVectorBase and put data in a union with some other types.
/jakob