[analyzer][RFC] Test all checkers on all tests


There is an idea for CSA testing. Currently each checker has it own test file(s) on which it is tested. Each checker has its own development history, and many of them were tested on code bases highly different from each other. That's why one checker may crash (i.e. produce assertion fail or UNREACHABLE) for code base on which it was never tested.

We tried to launch all checkers on all tests, by replacing `-verify' flags with `-analyzer-checker=<list of all packages>' through the lit.local.cfg (using config.substitutions.append method). A lot of crashes are obtained on our version of CSA (which is somehow different from upstream, however).

Moreover, we even tried Sema/SemaCXX tests and obtained several more crashes. Test files usually describe some corner cases, so this method will allow to test the analyzer on them.

I believe that such testing mode can be useful to improve the analyzer stability. Does anybody have any ideas, would it be appropriate? Should we add this mode to the default make check, or create a separate target for it?


I do like the idea of having a target for this but I would not make it part of the make check target. I think it would be a measurable overhead for the runtime of the tests and I am also not sure that we can/want to make that target clean. Maybe in some cases, the return on investment would be too low and we simply have other priorities or the side effect.



I think this is a great idea! I do think we’ll need to make sure not to run alpha checkers here since those often have assertion failures on purpose for some code constructs since they are work in progress.

I agree with Gabor that we should probably have this be a separate target — but we should have bots run it.


Tried this. Found a couple of "Check name is not set properly." assertion failures! Namely:

$ bin/clang -cc1 -internal-isystem lib/clang/7.0.0/include -nostdsysteminc -analyze -analyzer-checker=core,unix,osx,cplusplus,optin,nullability,valist,llvm -analyzer-checker=unix.cstring.BadSizeArg -analyzer-store=region -Wno-strncat-size -Wno-strlcpy-strlcat-size -Wno-sizeof-array-argument -Wno-sizeof-pointer-memaccess -verify ../llvm/tools/clang/test/Analysis/cstring-syntax.c

$ bin/clang -cc1 -internal-isystem lib/clang/7.0.0/include -nostdsysteminc -analyze -analyzer-checker=core,unix,osx,cplusplus,optin,nullability,valist,llvm -triple i386-apple-darwin10 -analyzer-checker=security.insecureAPI,security.FloatLoopCounter ../llvm/tools/clang/test/Analysis/security-syntax-checks.m -verify

That was it though, not much crashes on the whole Clang's test suite. I guess those guys who send us crashes from the GCC test suite did a really good job^^ (eg. https://reviews.llvm.org/D39800). Also i didn't enable alpha checkers.

I'd love to have this sort of stress test when i'm working with the AST that i don't fully understand. I guess it's a good idea for a buildbot (eg. add to StaticAnalyzerBenchmarks [Jenkins] or make a new one?).