You might be interested in this thread about potentially removing GEP from the IR: [RFC] Replacing getelementptr with ptradd
So if you want to do the offset calculations in integers, that’s fine, then do the pointer-plus-byte-offset with GEP i8.
(Some optimizations might not work as well with that, but those are arguably bugs in the optimizations, and if ptradd does happen, then those bugs will get fixed.)