From Mike Ash’s Blog post I learned that proper use of objc_msgSend requires you to cast the function so it accepts the given arguments (and to avoid implicit arguments’ type promotion). This, however, implies that we cast a function with signature of void objc_msgSend(void) to an arbitrary incompatible function pointer and then call it, which is UB according to the C standard:
6.3.2.3 Pointers/8: A pointer to a function of one type may be converted to a pointer to a function of another type and back again; the result shall compare equal to the original pointer. If a converted pointer is used to call a function whose type is not compatible with the referenced type, the behavior is undefined.
One thing what caught my eye is that when calling the function inline clang generates direct function call and doesn’t really use the pointer semantic:
objc_msgSend is not a function written in C; it’s part of the Objective-C runtime, and accepts a non-standard calling convention. Normal rules about “undefined behavior” don’t apply the same way.
As a practical matter, the LLVM IR generated for a real message send is exactly the same as that cast.
This is precisely why objc_msgSend cannot be implemented in C. It is in the same category as PLT stubs: trampolines that are guaranteed to forward their arguments to another function and return them (GCC has some extensions to try to support these but they don’t work on most targets).
From the perspective of the C abstract machine, the type of the function is dynamic. It is only UB if you call the function with the wrong types, but the type depends on the values of the first two arguments.
I’m sorry I didn’t quite understand this part: what exactly does make the function dynamic? And why the same is not true for regular C-functions “from perspective of the C abstract machine” then? I mean 6.3.2.3/8 doesn’t even impose requirements on implementation, whether the function is an assembly code or not, it merely says about function invocation, and it’s unclear at which point we can cancel the C standard and stop bothering about it.
The statement in the C spec is about the call site matching the implementation. In C, the types of the arguments are fixed in the implementation and so there is no possible way to write a non-variadic function that can be called with different types. The objc_msgSend function is not implemented in C and so the C standard has no impact on its implementation. The call site must follow the rule that the types that it provides match those expected by the caller, but in the case of objc_msgSend this type is not static (as it is for all functions implemented in C), but of the self and _cmd arguments.