fixing bugs fuzzed out of clang

Hi Clang devs,

In the new year I would like to ask you all to consider fixing clang bugs found by fuzzing (that includes, but is not limited to, https://llvm.org/bugs/show_bug.cgi?id=23057)

The existing fuzzer bot is reporting known bugs that are not being fixed for months.
E.g. http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fuzzer/builds/5328/steps/stage2%2Fasan%2Bassertions%20run%20clang-fuzzer/logs/stdio
This precludes us from treating these bugs as errors and make the bot red on regressions.

Also, these shallow bugs prevent us from finding deeper bugs with potential security implications, and there are some such.
E.g. the bug below means that no one can safely host clang as a web service.

echo “* a ((int () (o W, *&])) 0” | ./bin/clang -x c++ -

==13059==ERROR: AddressSanitizer: heap-use-after-free on address 0x61500000e538 at pc 0x00000081df99 bp 0x7ffdbdcb3630 sp 0x7ffdbdcb2de8

We (Intel clang team) will take a look and fix some of these.

Yours,
Andrey

Hi Kostya,

I would like to repro and fix some of the issues that fuzzed found but I don’t have system Clang on my machine but I do have Clang sources so I need to make Clang to have ASan and after that use that Clang to build Clang one more time but with ASan enabled. It looks like in Clang doc there is no info how to do it. I only found LLVM_USE_SANITIZER so I did it like this:

cd $ROOT
mkdir release

cd release
cmake …/llvm -DCMAKE_BUILD_TYPE=Release

make

cd …
mkdir build
cd build
CC=$ROOT/release/bin/clang CXX=$ROOT/release/bin/clang++ cmake …/llvm -DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_SANITIZER=Address
make

Last build seems to hang on compilation of llvm/lib/IR/Function.cpp. I break it after about 30min and 25G RAM consumed. Without -fsanitize=address compilation takes about 30 sec. Do you have bot that checks self build with ASan? Is it known issue?

Thanks,
Dmitry

Hi Kostya,

I would like to repro and fix some of the issues that fuzzed found

Nice!

but I don't have system Clang on my machine but I do have Clang sources so
I need to make Clang to have ASan and after that use that Clang to build
Clang one more time but with ASan enabled. It looks like in Clang doc there
is no info how to do it.

Mmmm.
These pages have some related discussions, but you probably don't need
them...
https://github.com/google/sanitizers/wiki/MemorySanitizerBootstrappingClang
https://github.com/google/sanitizers/wiki/AddressSanitizerHowToBuild

I only found LLVM_USE_SANITIZER so I did it like this:

cd $ROOT
mkdir release
cd release
cmake ../llvm -DCMAKE_BUILD_TYPE=Release
make
cd ..
mkdir build
cd build
CC=$ROOT/release/bin/clang CXX=$ROOT/release/bin/clang++ cmake ../llvm
-DCMAKE_BUILD_TYPE=Debug -DLLVM_USE_SANITIZER=Address
make

Last build seems to hang on compilation of llvm/lib/IR/Function.cpp.

I routinely build clang with clang+asan.
llvm/lib/IR/Function.cpp indeed takes lots of time, an order of 3 minutes,
but not even close to what you see.
Mostly likely the difference is that you use -DCMAKE_BUILD_TYPE=Debug
and I use -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
(in other words, I use -O2).
In general, we almost never use asan with -O0 because it'll be too slow.

BTW, I encourage you to switch from "make" to "ninja" -- works much faster
and nicer

I break it after about 30min and 25G RAM consumed.
Without -fsanitize=address compilation takes about 30 sec. Do you have bot
that checks self build with ASan?

Yes.
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap
You can check the exact commands e.g. here:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/10702/steps/build%20clang%2Fasan/logs/stdio
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON ...

Is it known issue?

I would guess what you see is a manifestation of
https://llvm.org/bugs/show_bug.cgi?id=17409,
which has been bothering us for years, but I have not checked this
particular case.

Thanks for doing this!

--kcc

Yes, it was due to using Debug build. With Release build I was able to build self build without hang on IR/Function.cpp. It was not a problem of using make vs ninja because I got clang command line and run it manually with the same result. I do prefer Debug build to be able to run debugger properly on sanitized binaries. Running GDB on asanified release binaries I found that debug info is weak and stepping over source code very often jumps to the begging on the function so it means that some instructions have no source info but I don’t know was it instructions inserted by asan instrumentation or not. Also it is very inconvenient that asan terminates execution when finds a problem instead of breaking into the debugger - it is my kind feature request :slight_smile:

Yes, it was due to using Debug build. With Release build I was able to
build self build without hang on IR/Function.cpp. It was not a problem of
using make vs ninja

My recommendation to use ninja is unrelated to your problem. It's just a
better build tool, imho.

because I got clang command line and run it manually with the same
result. I do prefer Debug build to be able to run debugger properly
on sanitized binaries.

Understood. You can probably build Function.cpp with -O2 manually and then
build the rest with -O0.

Running GDB on asanified release binaries I found that debug info is weak

Yep.

and stepping over source code very often jumps to the begging on the
function so it means that some instructions have no source info but I don't
know was it instructions inserted by asan instrumentation or not. Also
it is very inconvenient that asan terminates execution when finds a problem
instead of breaking into the debugger - it is my kind feature request :slight_smile:

See https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger
(I personally almost never use gdb with asan, so can't share any more
useful tricks)

--kcc

Thank you for the link, debugger tips in https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger are very useful but partially outdated. For example, it seems that AsanDie mentioned there is now __sanitizer::Die. That is why it would be much easier if asan detects attached debugger and does ASAN_OPTIONS=abort_on_error=1 under debugger. I think it will be user friendly behavior and would lower adoption bar :slight_smile:

Thank you for the link, debugger tips in
https://github.com/google/sanitizers/wiki/AddressSanitizerAndDebugger are
very useful but partially outdated. For example, it seems that AsanDie
mentioned there is now __sanitizer::Die. That is why it would be much
easier if asan detects attached debugger and does
ASAN_OPTIONS=abort_on_error=1 under debugger. I think it will be user
friendly behavior and would lower adoption bar :slight_smile:

I once asked for something like this in ASan group but folks weren't interested. You can take a look at https://github.com/yugr/libdebugme

BTW, -DCMAKE_BUILD_TYPE=Debug builds llvm/lib/IR/Function.cpp with -O1 and it hangs (or at least compile time is more that 40 min). On -DCMAKE_BUILD_TYPE=Release it is -O2 compilation and it works fine.

For some reason I can not reproduce the problem you have with Function.cpp:
cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DLLVM_USE_SANITIZER=Address $HOME/llvm

ninja opt

If you still see it, may I ask you to file a bug with preprocessed Function.cpp and exact clang command line?

For some reason I can not reproduce the problem you have with Function.cpp:
cmake -GNinja -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=clang
-DCMAKE_CXX_COMPILER=clang++ -DLLVM_USE_SANITIZER=Address $HOME/llvm
ninja opt

<builds fine>

If you still see it, may I ask you to file a bug with preprocessed
Function.cpp and exact clang command line?

Reid recently fixed this in r258897 and related commits.

-- Sean Silva

I confirm that -DCMAKE_BUILD_TYPE=Debug now builds successfully with ASan’itized compiler.