The more I follow LLVM, the more it seems like you in general just shouldn’t use LLVM structs. (The only counter-example I know is for multiple return values in registers.)
For example, [RFC] Replacing getelementptr with ptradd is moving away from using them for your second example. If your frontend has layout information already, you might consider just doing things like
%second_field_addr = getelementptr i8, ptr %ptr, i32 4
and avoiding the %my_struct.
(That’s what it looks like clang does, FWIW: https://cpp.godbolt.org/z/GMEc3b861.)