[Background: Currently we're trying to eliminate the concept of "transfer functions" from the static analyzer, which has gradually been supplanted by the Checker interface. The only implementation of the transfer functions, CFRefCount, has two roles: general-purpose function and method evaluator, and retain-count checking for the Cocoa and CoreFoundation frameworks. The former is being moved to ExprEngine*, the latter to a regular checker called RetainReleaseChecker. Of course, there are always design issues; I'm trying to make sure they get cleared up on-list one by one.]
In the process of eliminating CFRefCount from the analyzer, one of the issues is handling an Objective-C message whose result aliases the receiver.
- Currently TransferFuncs provides an evalObjCMessage hook, analogous to evalCall, but that's not really /ever/ going to be correct -- any message can be overridden by a subclass and have added functionality, so we don't want to expose that as a hook for checkers.
- Using a postObjCMessage hook is a problem because the order of invocation of checkers is not well-defined -- another checker could make use of a conjured return value before we mark that it's aliased. We could force RetainReleaseChecker to the front, but then we're just back to TransferFuncs in another guise. (And what happens when we need to force a second checker to the "front"?)
- Modelling arbitrary "a == b" constraints is a very hard problem when there may be existing info in a ProgramState about both a and b.
- Modelling "a == b" constraints where both types are pointers is a little better, but still not pleasant because of trying to merge /everything/ we might know about the symbolic MemRegions at the other ends of the pointers. This is more tractable but still quite hard (and would require many checkers to implement evalAssume, or maybe a new "substituteSymbol" callback).
- Modelling "a == b" constraints where both types are *Obj-C objects* is even better...but you can still access their instances variables, so not really.
Okay, hang on a second. The only methods that are receiver-aliasing are -retain and -autorelease (and technically, -self and -init). We have method families now. Can we just make ExprEngineObjC reason about method families to handle receiver aliasing for these four methods?
Pretty much all remaining handling of Obj-C method calls can be done using a standard preMessage or postMessage callback.