Kostya,
I think I’ve found what looks like a reproducible bug in libFuzzer. The code under test is built with ASan and the first ASan CHECK failure shows fuzzer in the stack trace. (see below)
One of the factors that may be unique in my testing is that each iteration can take a very long time to execute (tens or hundreds of seconds).
Let me know if you need more info, I think it shouldn’t take much test time to reproduce this.
================== Job 2 exited with exit code 256 ============
Flag: verbosity 3
Flag: use_traces 1
Flag: timeout 100
Flag: max_len 16384
Seed: 3259211893
PreferSmall: 0
#0 READ units: 4975 exec/s: 0
#1 pulse cov: 32410 bits: 30791 indir: 714 units: 4975 exec/s: 0
NEW0: 32410 L 13869
==31301==AddressSanitizer CHECK failed: /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467 “((n % 16)) == ((0))” (0x1, 0x0)
#0 0x11d3b7 in __asan::AsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/asan/asan_rtl.cc:67:3
#1 0x122f1f in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:159:5
#2 0x134317 in __sanitizer::CoverageData::Update8bitCounterBitsetAndClearCounters(unsigned char*) /home/brian/src/fuzzpy/llvm_src/llvm/projects/compiler-rt/lib/sanitizer_common/sanitizer_coverage_libcdep.cc:467:5
#3 0x1b7b53 in fuzzer::Fuzzer::PrepareCoverageBeforeRun() /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:264:5
#4 0x1b501b in fuzzer::Fuzzer::RunOne(std::vector<unsigned char, std::allocator > const&) /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:212:3
#5 0x1b6be3 in fuzzer::Fuzzer::ShuffleAndMinimize() /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerLoop.cpp:195:11
#6 0x14477b in fuzzer::FuzzerDriver(std::vector<std::string, std::allocatorstd::string > const&, fuzzer::UserSuppliedFuzzer&) /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:303:3
#7 0x14183f in fuzzer::FuzzerDriver(int, char**, fuzzer::UserSuppliedFuzzer&) /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:201:10
#8 0x141427 in fuzzer::FuzzerDriver(int, char**, int ()(unsigned char const, unsigned int)) /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerDriver.cpp:196:10
#9 0x1873e3 in main /home/brian/src/fuzzpy/llvm_src//llvm/lib/Fuzzer/FuzzerMain.cpp:19:10
#10 0xb6c86775 in __libc_start_main /build/buildd/glibc-2.21/csu/libc-start.c:289
DEATH:
artifact_prefix=‘./’; Test unit written to ./crash-ec9fa023e9db127e2589d0ab4c506055e4174611
Hi Brian,
Yes, looks like a bug in sanitizer coverage, please send the reproducer.
Kostya,
Here’s the git repo: https://bitbucket.org/ebadf/fuzzpy
I’ve only tested it on arm7 and x86_64 linux, I expect there’s a good chance it may not work on other OSs.
If you can build it successfully (“./build.sh”, requires clang and clang++ in your path), then you should run the “testemail” case like so:
while true; do ITERS=1000 ./run.sh tests/build/testemail tests/testemail/inputs/; done
Let me know if you have any challenges building or running the test case.
Kostya,
Here's the git repo: Bitbucket
I've only tested it on arm7 and x86_64 linux, I expect there's a good
chance it may not work on other OSs.
If you can build it successfully ("./build.sh", requires clang and clang++
in your path), then you should run the "testemail" case like so:
Does not build for me out of the box:
./build.sh: line 70: ./configure: No such file or directory
I wonder if a smaller test possible here.
Meanwhile, here is a workaround for you.
Instead of
SANITIZE_COV_OPTS="-fsanitize-coverage=bb,indirect-calls,8bit-counters"
try using
SANITIZE_COV_OPTS="-fsanitize-coverage=edge,indirect-calls"
while true; do ITERS=1000 ./run.sh tests/build/testemail
Ah, yes – you need to clone with --recursive.
I will try the workaround though.
I am able to build the fuzzer, but not to reproduce the crash.
First: you seem to be using a not-the-most-recent libfuzzer source, please update.
You will need to change LLVMFuzzerTestOneInput to return 0.
Second, for some reason, the fuzzing is very slow, about 1 exec/s.
You probably want to fix that too.
Third: does this code have threads? dlopen?
Running it for a short period does not show either, but probably the fuzzer discovers some such input that triggers dlopen in a non-main thread.
I guess the sanitizer coverage has a race in this case.
So, just drop 8bit-counters. (This flag is far from optimal, we may actually remove it completely at some point).
hth,
–kcc
Oh I thought it was really recent (1-2 weeks behind ToT ). I must have done something wrong there – I’ll check.
Yes, it’s extraordinarily slow and it may do dlopen (), I’ll find out.
No, I don’t believe it’s using other threads but I will check.
I would let it run for 24h before giving up on a crash though.
Oh I thought it was really recent (1-2 weeks behind ToT ). I must have
done something wrong there -- I'll check.
Yes, it's extraordinarily slow
and it may do dlopen (), I'll find out.
No, I don't *believe* it's using other threads but I will check.
I would let it run for 24h before giving up on a crash though.
This makes little sense because with such a rare reproducer it's hard to
fix the bug.
So far my only idea is a race between libFuzzer's actions in the main
thread and dlopen in a non-main thread.
And this is not the code I want to keep long term, so I am reluctant to fix
it now. 
And you have a workaround 