I've been playing with afl-fuzz to fuzz the clang test suite. In
the first 11 hours I have discovered 34 distinct assertion failures in
clang -std=c++11 and at least one segmentation fault (I didn't yet do
anything to tell different SEGVs apart), all on a recent HEAD. I
thought I'd share the methodology and the initial findings with you. I
haven't reported any bugs yet; feel free to do so if you are so
inclined. I might do that later.
For the 35 minimized test cases and their outputs, see , or
download a tarball from .
The git versions I'm running the tests against are:
llvm 1610d6e Add missing FP build attribute tests.
clang 64a12ad Driver: Objective-C should respect -fno-exceptions
compiler-rt d6e5390 [ASan, LSan] Improve tracking of thread creation.
Some quick notes:
* These have been generated using afl-fuzz 0.85b using the clang test
suite as input.
* The test suite has been first minimized using afl-fuzz's
* Finding the first 35 crashes (34 distinct assertion failures and at
least one segfault as I didn't yet do anything to tell them apart)
took about 11 hours on a Core i7-2600.
* afl-fuzz is a directed fuzzing tool. When it discovers an input that
exercises new edges in the binary, it adds it to the queue as a new
input for fuzzing. After 11 hours the fuzzer is at most 0.05%
through its queue. A small cluster would be nice for this. I suspect
it would find new crashes after weeks of fuzzing on a binary of this
* If you wish to try it yourself, note the following:
* Increase MAP_SIZE_POW2 in afl-fuzz's config.h.
* 17 is not sufficient, 18 has looked good for the first 11 hours...
* I built LLVM with -DLLVM_ENABLE_ASSERTIONS=ON
-DLLVM_ENABLE_CRASH_OVERRIDES=OFF (and of course using afl-clang as
* Running clang -cc1 directly with the input and with gives me about
50-60 executions per second on each of the 8 instances (for a
total of 400-480/s) instead of 5-6/s.
* Use something like -ferror-limit 5 -Werror.
* You might want to experiment with even smaller error limit.
* Using ASAN might make sense too (afl supports it quite nicely, at
least on 32-bit).
* I consider it hideous how well afl-fuzz works for a tool which does
essentially sed-level magic on compiler-generated assembly to
instrument the program.