RFC: Generalize means the sanitizers work with memory

Ivan_A_Kosarev · February 23, 2017, 6:16pm

Overview

sanitizers-instrumentation.diff (10.3 KB)

Kostya_Serebryany · February 28, 2017, 1:32am

Hi Ivan,

I’ve seen your message, but did not have a chance to carefully read, sorry. Busy weeks.
I may have time next week, or maybe some one else replies earlier.
Don’t hesitate to ping me ~ mid next week.

Some suggestions:

if you use http://llvm.org/docs/Phabricator.html for patches you are more likely to get attention from us.
be more concrete, e.g. instead of “platforms that lack such support” mention which exactly
platforms are affected (and what is the rest of the LLVM support story for them)

_sean_silva · February 28, 2017, 2:59am

+Hal

IIRC, Hal mentioned that he did something like this for a no-MMU HPC environment he was working in.

– Sean Silva

Finkel_Hal_J · March 9, 2017, 1:58pm

Hi Ivan,

Thanks for posting this; I’m excited by this proposal - if we can get this kind of support in without making the implementation non-trivially-harder to maintain, that would be a positive development. As Sean mentioned, I did something along these lines to adapt ASan to the IBM BG/Q - an HPC system that uses a lightweight operating system. On the BG/Q, the lightweight operating system does support virtual memory for some special-purpose mappings, but it does not support mapping unreserved pages (i.e. MAP_NORESERVE is not supported, and this functionality is not supported any other way). As a result, the mechanism that the sanitizers use to cover the complete address space using shadow memory - by mapping a large region of unreserved pages - won’t work in this environment. Systems without virtual memory at all will obviously have the same problem: All shadow memory must be physically backed. I’ll also mention that many normal Linux HPC environments are configured with overcommit turned off, and I believe that using the sanitizers in such environments would also currently not work.

Because all shadow memory must be physically backed, it must be allocated judicially, and the mapping process might need to be more complicated than a simple shift/offset. On the BG/Q, there were a few distinct regions of virtual memory that needed to be mapped into a single shadow region in the part of the address space where heap allocations could be made - as a result, I used a more-complicated mapping function.

In this light, I’m trying to understand your proposal. I see that you’re proposing to add support for some kind of additional translation scheme between virtual addresses and physical addresses, but I’m not exactly sure how you propose to use them. It might help if you were to provide some hypothetical implementation of these translations for a simple system so that we can understand the usage model better. I’d also like to better understand how the instrumentation works; if the mapping always replaced by these __asan_mem_to_vshadow/__asan_mem_to_pshadow calls?

Finally, I recommend that we layer this support so that we have:

[regular system] → [system without (sufficient) unreserved pages] → [system without any mmu]

I’d like a clear explanation of how these last two differ. It looks like you have support for manually zeroing pages for the last category. Please explain exactly how this scheme works.

Thanks,

Hal

Ivan_A_Kosarev · March 13, 2017, 7:32pm

Hello Hal,

Thanks a lot for your feedback. Particularly, I appreciate mentioning HPC systems as potential targets for this work as it helps with figuring out what the generalized memory interface would look like.

Answering your questions: what we propose with this RFC is to support platforms for which there is no way to adopt sanitizers by adding a special mapping or tweaking the MEM_TO_SHADOW() macro. As you already said, the problem with such platforms is that the amount of available physical memory is not sufficient to shadow-map the whole address space to sanitize. The solution we propose is to claim a fixed set of ways, such as macros and functions, that provide access to physical shadow memory so that by implementing these macros and functions one can support sanitizers even on platforms that: 1) do not have hardware support for virtual memory and 2) can only allocate physical memory by relatively small pieces whose base addressses are not known at compile-time. This includes implementing a software shadow memory manager on top of a malloc()-like API. In addition, the resulting support shall be compact and efficient enough to be practical on such platforms and the introduced abstraction layer shall have zero penalty in terms of code space and performance for the already supported targets.

The proposed approach to the abstraction layer is to provide macros and functions that perform necessary operations with physical shadow memory by given virtual shadow addresses or virtual shadow address ranges. For example, for Asan there is function VShadowToPShadow() declared as:

u64 *VShadowToPShadow(uptr vs);

that returns a pointer to a physical shadow cell by its virtual address and makes sure that the piece of physical shadow memory (the physical shadow page) the address belongs to is allocated and accessible. For perfomance reasons there are also block shadow memory functions that peform various operations over virtual address ranges rather than individual addresses. There is also a function that explicitly releases physical shadow memory. That function can be implemented in any way suitable for a given specific platform. The only requirement is that subsequent read accesses to the released shadow memory yield zeros, so the simplest implementation is zeroing out the specified region.

Please see the updated patch at:

for details. This patch implements a software shadow memory manager on top of Linux mmap(). With this patch we pass Asan and Tsan tests with a promising slowdown ratio. Re: instrumentation: yes, to support platforms that only support manual/explicit allocation of physical memory the only way is to instrument the code to sanitize with RTL calls. One important quality of the abstraction layer that we would like to maintain is that it is never required to perform backward physical-to-virtual translations as they may be extremely inefficient in some cases. Since the santiziers themselves require shadow-to-application memory translations to be supported, we have to deal with both the concepts of virtual and physical shadow addresses. Since the abstract layer operates in terms of virtual shadow addresses, it does not affect how application memory addresses translate to virtual shadow addresses. This means one can choose whatever mapping will do better for his platform and then decide whether to rely on hardware-driven allocation of physical pages or implement a custom software memory manager. Thanks again and please let me know if I can help more. Regards, Ivan