Porting ASan to AArch64

Project Infrastructure LLVM Dev List Archives
#1

Hello,

I have been working on porting ASan to AArch64. I am building compiler-rt in “standalone mode” targeting aarch64. My build is successful, but I get the following runtime error when I run an ASan enabled executable through qemu-aarch64:

==29184==Parsed ASAN_OPTIONS: verbosity=1
==29184==AddressSanitizer: failed to intercept ‘__isoc99_printf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_sprintf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_snprintf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_fprintf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_vprintf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_vsprintf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_vsnprintf’
==29184==AddressSanitizer: failed to intercept ‘__isoc99_vfprintf’
==29184==AddressSanitizer: libc interceptors initialized

[0x002000000000, 0x007fffffffff] || HighMem ||
[0x001400000000, 0x001fffffffff] || HighShadow ||
[0x001200000000, 0x0013ffffffff] || ShadowGap ||
[0x001000000000, 0x0011ffffffff] || LowShadow ||
[0x000000000000, 0x000fffffffff] || LowMem ||
MemToShadow(shadow): 0x001200000000 0x00123fffffff 0x001280000000 0x0013ffffffff
redzone=16
max_redzone=2048
quarantine_size=256M
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 1000000000
==29184==ERROR: AddressSanitizer failed to allocate 0xc00000000 (51539607552) bytes at address 1400000000 (errno: 12)
==29184==ReserveShadowMemoryRange failed while trying to map 0xc00000000 bytes. Perhaps you’re using ulimit -v

This appears to be the same issue discussed in this thread:
https://code.google.com/p/address-sanitizer/issues/detail?id=246#makechanges

I contacted Christophe Lyon, and he said he had successfully ported ASan to aarch64 and had committed a patch to enable this to LLVM:
https://github.com/llvm-mirror/compiler-rt/commit/33465467e7f8243933100bab3cf8f34b79a8f54c

His changes from this patch still appear to be configured correctly in my build, so I am not sure why I am getting this error. Is anyone able to shed some light on this?

Thanks,
-Gideon

#2

+Greg in case he’s seen similar problems.

Basically, ASan is unable to map the necessary shadow memory. You may want to dump the contents of /proc/self/maps and look if it conflicts
with the range ASan is trying to use. errno 12 is ENOMEM. Are you sure you don’t have constraints on the virtual memory your process can use?

#3

The mmap error is an issue with the qemu-aarch64 model I’ve been using. I am not sure how to get a dump of /proc/self/maps, as the model is configured as a non-interactive elf interpreter. I can successfully mmap a smaller address range, and I tried configuring the environment using setrlimit(), but I still get the error for the shadow memory range. I may try to debug the qemu model further, but for now, I have switched to using a different interactive model, and the mmap now succeeds. However, I am getting a different error. When I try to run a simple “Hello World” program that prints using cout, I get this segfault error:

==69==Parsed ASAN_OPTIONS: verbosity=2
==69==AddressSanitizer: failed to intercept ‘__isoc99_printf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_sprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_snprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_fprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vsprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vsnprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vfprintf’
==69==AddressSanitizer: libc interceptors initialized

[0x002000000000, 0x007fffffffff] || HighMem ||
[0x001400000000, 0x001fffffffff] || HighShadow ||
[0x001200000000, 0x0013ffffffff] || ShadowGap ||
[0x001000000000, 0x0011ffffffff] || LowShadow ||
[0x000000000000, 0x000fffffffff] || LowMem ||
MemToShadow(shadow): 0x001200000000 0x00123fffffff 0x001280000000 0x0013ffffffff
redzone=16
max_redzone=2048
quarantine_size=256M
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 1000000000
==69==Installed the sigaction for signal 11
==69==SetCurrentThread: 0x007fb7ff1000 for thread 0x007fb7ff6000
==69==T0: stack [0x007fff800000,0x008000000000) size 0x800000; local=0x007ffffffbd8
==69==AddressSanitizer Init done
ASAN:SIGSEGV

#4

Basically, ASan is unable to map the necessary shadow memory. You may want
to dump the contents of /proc/self/maps and look if it conflicts
with the range ASan is trying to use. errno 12 is ENOMEM. Are you sure you
don't have constraints on the virtual memory your process can use?

Similar thread in GCC ML: https://gcc.gnu.org/ml/gcc/2014-06/msg00023.html .
It looks like QEMU-user does not yet support the NORESERVE flag.

-Y

#5

Similar thread in GCC ML: https://gcc.gnu.org/ml/gcc/2014-06/msg00023.html .
It looks like QEMU-user does not yet support the NORESERVE flag.

Thanks! Applying this patch to qemu 2.0 fixed the mmap issue:
http://lists.gnu.org/archive/html/qemu-devel/2014-02/msg00319.html

I am still getting the segfault error, though.

#6

Cool, making progress. :slight_smile:

  Expected Passes : 43
  Expected Failures : 2
  Unsupported Tests : 84
  Unexpected Failures: 138

For that segfault, can you run in gdb and get a stack trace?

Thanks,
Greg

#7

When I try to run a simple “Hello World” program that prints using cout, I get this segfault error:

==69==Parsed ASAN_OPTIONS: verbosity=2
==69==AddressSanitizer: failed to intercept ‘__isoc99_printf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_sprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_snprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_fprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vsprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vsnprintf’
==69==AddressSanitizer: failed to intercept ‘__isoc99_vfprintf’
==69==AddressSanitizer: libc interceptors initialized

[0x002000000000, 0x007fffffffff] || HighMem ||
[0x001400000000, 0x001fffffffff] || HighShadow ||
[0x001200000000, 0x0013ffffffff] || ShadowGap ||
[0x001000000000, 0x0011ffffffff] || LowShadow ||
[0x000000000000, 0x000fffffffff] || LowMem ||
MemToShadow(shadow): 0x001200000000 0x00123fffffff 0x001280000000 0x0013ffffffff
redzone=16
max_redzone=2048
quarantine_size=256M
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 1000000000

==69==Installed the sigaction for signal 11
==69==SetCurrentThread: 0x007fb7ff1000 for thread 0x007fb7ff6000
==69==T0: stack [0x007fff800000,0x008000000000) size 0x800000; local=0x007ffffffbd8
==69==AddressSanitizer Init done
ASAN:SIGSEGV

==69==ERROR: AddressSanitizer: SEGV on unknown address 0x100fffffff6e (pc 0x0000004a37b4 sp 0x007ffffffb70 bp 0x007ffffffbe0 T0)
==69==AddressSanitizer CHECK failed: /local/mnt/workspace/gideonb/projects/toolchains/open-draco/compiler-rt/lib/sanitizer_common/sanitizer_stacktrace.cc:68 “((count)) < ((size))” (0x1, 0x1)

I’ve been using gdb to debug the segfault error I get when I run an ASan enabled binary:

==6038==ERROR: AddressSanitizer: SEGV on unknown address 0x000000100022 (pc 0x0000004a3b8c sp 0x0040008000b0 bp 0x004000800170 T0)

gdb gives the following error:

Program received signal SIGSEGV, Segmentation fault.
0x00000000004a3b8c in main () at test.cpp:6
6 {

I’ve stepped through ASan initialization code, and there appear to be no errors until main() is called. I did a back trace and a dump of the assembly from this point:

(gdb) bt
#0 0x00000000004a3b8c in main () at test.cpp:6
#1 0x0000004000aa0288 in __libc_start_main (main=0x0, argc=0, argv=0x4000800180,
init=, fini=, rtld_fini=,
stack_end=) at libc-start.c:285
#2 0x00000000004a39ac in _start ()
(gdb) x/10i $pc-20
0x4a3b78 <main()+160>: lsr x11, x8, x11
0x4a3b7c <main()+164>: orr x12, xzr, #0x100000000000
0x4a3b80 <main()+168>: orr x11, x11, x12
0x4a3b84 <main()+172>: mov x12, #0x0 // #0
0x4a3b88 <main()+176>: add x12, x11, x12
=> 0x4a3b8c <main()+180>: str x10, [x12]
0x4a3b90 <main()+184>: orr x10, xzr, #0x8
0x4a3b94 <main()+188>: add x10, x11, x10
0x4a3b98 <main()+192>: str w9, [x10]
0x4a3b9c <main()+196>: orr x10, xzr, #0x3
(gdb) p/x $x12
$8 = 0x100800100022
(gdb) p/x $x11
$9 = 0x100800100022

I noticed that the address reported in the ASan segfault error, 0x000000100022, is the same as the address held by x12, 0x100800100022, but with the top bytes chopped off. I think this makes sense, since the AArch64 address space is only 39-bits and ends at 0x7fffffffff. However, I have not been able to determine where the address stored in x12 is originating. Do you have any idea where this might be coming from?

Thanks,
-Gideon