I am trying to implement a checker for Clang Static Analyzer which needs to follow function calls. In some testcases, some calls can be nested to facilitate recursion. I wanted to understand how the Static Analyzer handles recursion (or cyclic function calls, for e.g. f1 calls f2, f2 calls f3, f3 calls back f1). Is there a limit to which the Static Analyzer will keep hopping into the next function before identifying a cycle and skipping the next function call to evaluate the rest of the function body?
There’s a limit of 4 nested calls. It doesn’t care whether it’s recursion or just nested calls. That’s mostly because our interprocedural analysis is explosive enough in complexity as it is to go any deeper. We’re also making an exception for “small” functions that don’t introduce any branches, these can go much deeper (branches are important because every branch doubles our analysis time as we have to explore both directions).
In that case, is there any way to skip to checking the other parts of the code, which are neglected because they are not reachable in the current nested call? For e.g. consider a recursive function f(int a), which calls f(a-1) somewhere in its body. The base case occurs when a = 1. Now, say we start the analysis with f(10), then because of the limit on depth of nesting, we will never reach the call to f(1), due to which static analyzer will rule out the base case as an unreachable part of the code, which is certainly not the case. Is there any work around to this?
One possible solution which I came up with was to check only the function definitions and not the calls themselves (so that we don’t bias any part of the code due to argument values. But, I have neither found any way to direct the Static Analyzer to prevent analyzing a function/method nor make it so that it checks a function/method twice.