AddressSanitizer flags in LLVM 3.5

Hello, LLVM developers

I would like to ask about the changes in the "nonpublic"
AddressSanitizer flags in llvm 3.5.

Prior to 3.5 (namely, on 3.4.2) there was a compile-time flag to
control the shadow offset. For example, one could do the following to
force the large offset:
-mllvm -asan-mapping-offset-log=44

Now, it has been removed and on x86_64 linux the smaller offset is always used.

Our use case for the large offset was using NVIDIA OpenGL/CUDA drivers
which apparently mmap certain memory ranges and fail when the
requested address does not match the return value of mmap. Using a
larger offset allows to work around the problem and use both the
binary driver and ASan which is very useful.

Could someone explain the reason behind forcing a small shadow offset
on 64-bit linux and add a public documented option to control it? It
is also interesting that OS X and FreeBSD use larger offset.

Meanwhile, those who are interested in using ASan and NVIDIA, can edit
"lib/Transforms/Instrumentation/AddressSanitizer.cpp" and replace
"Mapping.Offset = kSmallX86_64ShadowOffset" with "Mapping.Offset =
kDefaultShadowOffset64".

Hello, LLVM developers

I would like to ask about the changes in the "nonpublic"
AddressSanitizer flags in llvm 3.5.

Prior to 3.5 (namely, on 3.4.2) there was a compile-time flag to
control the shadow offset. For example, one could do the following to
force the large offset:
-mllvm -asan-mapping-offset-log=44

Now, it has been removed and on x86_64 linux the smaller offset is always
used.

Our use case for the large offset was using NVIDIA OpenGL/CUDA drivers
which apparently mmap certain memory ranges and fail when the
requested address does not match the return value of mmap. Using a
larger offset allows to work around the problem and use both the
binary driver and ASan which is very useful.

Could someone explain the reason behind forcing a small shadow offset
on 64-bit linux

The reason is performance and code size -- both get better by 3%-5% with
the small offset.

and add a public documented option to control it?

Oh my... This is not as trivial as it sounds.
We removed this option because (as we thought) no one was using it and
because it was actually hard to maintain.
This is a compile-time option, which uses a different ABI, and objects
built with different flags are incompatible.
GCC variant of asan does not support it either and GCC folks explicitly
said that they are not going to.

I don't have a good solution. Others?
In the meantime I would strongly suggest to try fixing the issue on the
NVIDIA side.

It

is also interesting that OS X and FreeBSD use larger offset.

Right. The small offset didn't work there for some reasons...

Meanwhile, those who are interested in using ASan and NVIDIA, can edit
"lib/Transforms/Instrumentation/AddressSanitizer.cpp" and replace
"Mapping.Offset = kSmallX86_64ShadowOffset" with "Mapping.Offset =
kDefaultShadowOffset64".

you will also need to change the run-time library.

--kcc

Hi Alexander, Kostya,

I’m actually one of that flag’s clients, but not for long.
It’s actually handy when bringing up ASan in a new environment, but the shadow offset is one of the first things that will be fixed and the flexibility of having a flag might give people the idea that it’s something that’s easy to change by simply changing the flag.

It might also mess up performance. We’re using a large-ish offset when testing (though not as large as 2^44) and we get an extra instruction + reg used because of that (mov shadow_offset, reg + or shadow_offset_reg, shadow_index_reg). It’s not ideal, but we’re still actively porting this.

Eventually, changing the shadow offset is not that hard. All it should take is change the appropriate version in the ASan pass and asan_mapping.h.

Since the flag is of very limited use (comes in handy when starting to port ASan, at most), I don’t think it’s a very big value to have it, either.

Alexander: Can you change your compiler so the ASan pass knows what offset you want to use for your use-case (by adding a triple to getShadowMapping() in AddressSanitizer.cpp, for example)?

Regards,

Filipe

FYI we'll probably have it for KAsan to simplify experiments with kernel memory layouts.

Hi, Kostya. Thanks for replying

Oh my… This is not as trivial as it sounds.
We removed this option because (as we thought) no one was using it and because it was actually hard to maintain.
This is a compile-time option, which uses a different ABI, and objects built with different flags are incompatible.
GCC variant of asan does not support it either and GCC folks explicitly said that they are not going to.

Thank for this. Now it makes sense. I did not actually think of it myself.

I don’t have a good solution. Others?
In the meantime I would strongly suggest to try fixing the issue on the NVIDIA side.

Unfortunately I am not working at NVIDIA so I cannot change that. But at work we’re using Nix package manager from NixOS so patching llvm, clang and rebuilding all the packages is trivial though a bit time consuming.

It
is also interesting that OS X and FreeBSD use larger offset.

Right. The small offset didn’t work there for some reasons…

Meanwhile, those who are interested in using ASan and NVIDIA, can edit
“lib/Transforms/Instrumentation/AddressSanitizer.cpp” and replace
“Mapping.Offset = kSmallX86_64ShadowOffset” with “Mapping.Offset =
kDefaultShadowOffset64”.

you will also need to change the run-time library.

Sorry, do you mean libc or what runtime? I will take a deeper look into llvm and clang source to make sure I didn’t break anything while patching our local tree. Basically we’re now building our project with ASan and the rest of system is not instrumented, so I think we don’t have to worry about using different offset sizes until we start instrumenting dependencies, right?