Hello ASan devs,
I am trying to improve ASan functionality with our D compiler, LDC. One of the main issues is that D has a garbage collector, and so I have to modify both the ASan instrumentation and the D runtime such that they work well together.
My question is about the FakeStack instrumentation to detect stack-use-after-return bugs.
The D garbage collector needs to scan the stack for pointers, in order to make sure that an object is no longer referenced before it can be collected. So when ASan is dynamically allocating a FakeStack, we have to inform the GC such that it scans that region for pointers too.
I wrote a pass that adds calls to
gc_removeRange for each encountered
__asan_stack_free, respectively. However, for small stack frames, LLVM does not emit a call to
__asan_stack_free but instead emits directly inlined equivalent IR, and so my pass misses a lot of stack frees and doesn’t work.
How can I get things to work?
I see a few possibilities:
- Add an LLVM option to disable the
__asan_stack_freeinlining, and use my pass.
- Add an LLVM option to emit calls to frontend-specified functions after
__asan_stack_free, which replaces my pass.
- Figure out how to use ASan’s
ForEachExtraStackRange(callback for each fakestack) and use that in D’s GC implementation.
Probably option 3 is the best (
gc_addRange involves mutex locking), but the hardest.
(I also noticed that I have to call __sanitizer_start_switch_fiber and __sanitizer_finish_switch_fiber upon fiber switches, but after reading the code, I think it’s only needed when FakeStack is enabled)
Thanks for your advice,