-fsanitize=memory failing on 3.9.0

Project Infrastructure LLVM Dev List Archives
1

I've compiled REALEASE_390/final but all "ninja check-msan" tests are
failing (http://lists.llvm.org/pipermail/llvm-dev/2016-September/104609.html)
I'm waiting for an account to be created to file a bug, but in the
mean time I thought I'd take a look at it myself.

My system is an Arch Linux system that is up to date as of this morning:
$ uname -a
Linux wink-desktop 4.7.2-1-ARCH #1 SMP PREEMPT Sat Aug 20 23:02:56
CEST 2016 x86_64 GNU/Linux

The installed compilers are:
$ pacman -Q clang clang-tools-extra gcc-multilib gcc-libs-multilib
clang 3.8.1-1
clang-tools-extra 3.8.1-1
gcc-multilib 6.1.1-5
gcc-libs-multilib 6.1.1-5

Reid Kleckner (http://lists.llvm.org/pipermail/llvm-dev/2016-September/104610.html)
speculates:
"There is probably some environmental issue on your system that makes
shadow memory allocation fail, or causes an early shadow memory
access."

I agree because I see similar startup errors on both clang 3.8.1
installed via arch linux and 3.9.0 I created.

Here is the trivial test program:
$ cat a.c
int main() {
  return 0;
}

Here is the compilation:
$ /home/wink/foss/llvm.3.9.0/build/bin/clang -fsanitize=memory -g -O0
-fno-omit-frame-pointer a.c -o a

And here is running it with gdb:
$ gdb a
GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;\.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;\.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a...done.
(gdb) run
Starting program: /home/wink/foss/llvm.3.9.0/test-msan/a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
__sanitizer::SizeClassAllocator64<123145302310912ul, 8796093022208ul,
8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>::AllocateBatch (
    this=this@entry=0x21289a0 <__msan::allocator>,
stat=stat@entry=0x2128970 <__msan::fallback_allocator_cache+109392>,
c=c@entry=0x210de20 <__msan::fallback_allocator_cache>,
class_id=class_id@entry=5)
    at ../projects/compiler-rt/lib/msan/../sanitizer_common/sanitizer_allocator.h:357
357 Batch *b = region->free_list.Pop();
(gdb) bt
#0 __sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>::AllocateBatch (
    this=this@entry=0x21289a0 <__msan::allocator>,
stat=stat@entry=0x2128970 <__msan::fallback_allocator_cache+109392>,
c=c@entry=0x210de20 <__msan::fallback_allocator_cache>,
class_id=class_id@entry=5)
    at ../projects/compiler-rt/lib/msan/../sanitizer_common/sanitizer_allocator.h:357
#1 0x0000000000443567 in
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback> >::Refill (this=this@entry=0x210de20
<__msan::fallback_allocator_cache>,
allocator=allocator@entry=0x21289a0 <__msan::allocator>,
class_id=class_id@entry=5)
    at ../projects/compiler-rt/lib/msan/../sanitizer_common/sanitizer_allocator.h:1003
#2 0x0000000000442af5 in
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback> >::Allocate (class_id=<optimized out>,
allocator=0x21289a0 <__msan::allocator>, this=0x210de20
<__msan::fallback_allocator_cache>)
    at ../projects/compiler-rt/lib/msan/../sanitizer_common/sanitizer_allocator.h:952
#3 __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>,
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback> >,
__sanitizer::LargeMmapAllocator<__msan::MsanMapUnmapCallback>

::Allocate (check_rss_limit=false, cleared=false, alignment=8,

size=<optimized out>, cache=0x210de20
<__msan::fallback_allocator_cache>, this=0x21289a0
<__msan::allocator>)
    at ../projects/compiler-rt/lib/msan/../sanitizer_common/sanitizer_allocator.h:1324
#4 __msan::MsanAllocate (zeroise=false, alignment=8, size=73,
stack=0x7fffffffcea0) at
../projects/compiler-rt/lib/msan/msan_allocator.cc:125
#5 __msan::MsanReallocate (stack=stack@entry=0x7fffffffcea0,
old_p=old_p@entry=0x0, new_size=new_size@entry=73,
alignment=alignment@entry=8, zeroise=zeroise@entry=false)
    at ../projects/compiler-rt/lib/msan/msan_allocator.cc:180
#6 0x000000000044475e in __interceptor_malloc (size=73) at
../projects/compiler-rt/lib/msan/msan_interceptors.cc:931
#7 0x00007ffff7de9161 in _dl_signal_error () from /lib64/ld-linux-x86-64.so.2
#8 0x00007ffff7de9323 in _dl_signal_cerror () from /lib64/ld-linux-x86-64.so.2
#9 0x00007ffff7de40be in _dl_lookup_symbol_x () from
/lib64/ld-linux-x86-64.so.2
#10 0x00007ffff7016db1 in do_sym () from /usr/lib/libc.so.6
#11 0x00007ffff74ae014 in ?? () from /usr/lib/libdl.so.2
#12 0x00007ffff7de93a4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#13 0x00007ffff74ae521 in ?? () from /usr/lib/libdl.so.2
#14 0x00007ffff74ae068 in dlsym () from /usr/lib/libdl.so.2
#15 0x00000000004193cc in __interception::GetRealFunctionAddress
(func_name=func_name@entry=0x499bb8 "__isoc99_printf",
func_addr=func_addr@entry=0x2b298d8
<__interception::real___isoc99_printf>,
    real=real@entry=4591392, wrapper=wrapper@entry=4591392) at
../projects/compiler-rt/lib/interception/interception_linux.cc:23
#16 0x0000000000476a5f in InitializeCommonInterceptors () at
../projects/compiler-rt/lib/msan/../sanitizer_common/sanitizer_common_interceptors.inc:5902
#17 __msan::InitializeInterceptors () at
../projects/compiler-rt/lib/msan/msan_interceptors.cc:1471
#18 0x000000000043f4c5 in __msan_init () at
../projects/compiler-rt/lib/msan/msan.cc:386
#19 0x000000000048d586 in msan.module_ctor ()
#20 0x000000000048d5dd in __libc_csu_init ()
#21 0x00007ffff6f18220 in __libc_start_main () from /usr/lib/libc.so.6
#22 0x00000000004192da in _start ()
(gdb)

When I compile it with clang 3.8.1 it fails in a different spot but
still in __interception::GetRealFunctionAddress:

$ clang -fsanitize=memory -g -O0 -fno-omit-frame-pointer a.c -o a
wink@wink-desktop:~/foss/llvm.3.9.0/test-msan
$ gdb a
GNU gdb (GDB) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;\.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/&gt;\.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from a...done.
(gdb) run
Starting program: /home/wink/foss/llvm.3.9.0/test-msan/a
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
0x000000000041e3b5 in
__sanitizer::SizeClassAllocator64<123145302310912ul, 8796093022208ul,
8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>::AllocateBatch(__sanitizer::AllocatorStats*,
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback> >*, unsigned long) ()
(gdb) bt
#0 0x000000000041e3b5 in
__sanitizer::SizeClassAllocator64<123145302310912ul, 8796093022208ul,
8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>::AllocateBatch(__sanitizer::AllocatorStats*,
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback> >*, unsigned long) ()
#1 0x000000000041e477 in
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<123145302310912ul,
8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>

::Refill(__sanitizer::SizeClassAllocator64<123145302310912ul,

8796093022208ul, 8ul, __sanitizer::SizeClassMap<17ul, 128ul, 16ul>,
__msan::MsanMapUnmapCallback>*, unsigned long) ()
#2 0x000000000041d9d1 in
__msan::MsanReallocate(__sanitizer::StackTrace*, void*, unsigned long,
unsigned long, bool) ()
#3 0x000000000041f8fe in malloc ()
#4 0x00007ffff7de9161 in _dl_signal_error () from /lib64/ld-linux-x86-64.so.2
#5 0x00007ffff7de9323 in _dl_signal_cerror () from /lib64/ld-linux-x86-64.so.2
#6 0x00007ffff7de40be in _dl_lookup_symbol_x () from
/lib64/ld-linux-x86-64.so.2
#7 0x00007ffff7016db1 in do_sym () from /usr/lib/libc.so.6
#8 0x00007ffff74ae014 in ?? () from /usr/lib/libdl.so.2
#9 0x00007ffff7de93a4 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
#10 0x00007ffff74ae521 in ?? () from /usr/lib/libdl.so.2
#11 0x00007ffff74ae068 in dlsym () from /usr/lib/libdl.so.2
#12 0x0000000000465c0c in __interception::GetRealFunctionAddress(char
const*, unsigned long*, unsigned long, unsigned long) ()
#13 0x000000000044f5e5 in __msan::InitializeInterceptors() ()
#14 0x000000000041a305 in __msan_init ()
#15 0x0000000000485be6 in msan.module_ctor ()
#16 0x0000000000485c3d in __libc_csu_init ()
#17 0x00007ffff6f18220 in __libc_start_main () from /usr/lib/libc.so.6
#18 0x0000000000418b4a in _start ()
(gdb)

Further more, there is a bug reported concerning a seg fault when
using msan on Arch Linux (FS#50385 : [clang] fsanitize=memory Segmenation fault) so
I'm not the only person in the world have a problem.

Any suggestions on what the problem might be?

2

I'm having the same problem on trunk. The last time I did run the
sanitizer tests was about a month or two ago, just before the release,
and it was working well.

A short bisect could tell us which commit broke it... I'll try
something on my side.

cheers,
--renato

3

Please do the bisect, nice to hear it recently worked!

At this point I built mean with debug and its failing with initializing a LFStack doing an atomic operation, IIRC.

4

So, I went as far back as 3 months ago and it still crashes. I think
Reid is right that some environment change (kernel version?) made it
crash.

Best thing to do is to put all that info into LLVM bugzilla, link all
relevant information and go from there.

Unfortunately, I'm not an expert on either msan or x86, so it would be
good to include the rest of the sanitizer folks on the bug. If you
create one, copy me and I'll try to put all relevant people in.

cheers,
--renato

5

If you are on glibc-2.24, did you patch it with the fix
24e2b1cede1952d7d4411a3cafd25dd8593dab9f that revert commits
80f87443eed17838fe453f1f5406ccf5d3698c25 and
a824d609581d5ee7544aabcbbc70e8da44b2b5b6? I had to do that since it
broke go, gcc, and clang address sanitizers without the patch.

6

I didn't. Shouldn't the arch glibc maintainer do that and push an update?

The bug FS#50385 : [clang] fsanitize=memory Segmenation fault doesn't seem to be
associated with the glibc package...

--renato

7

Open Mandriva seems to have a similar problem with glibc-2.24,
seg-faulting on MSAN's Linux/forkpty.cc.

Bero, have you reverted the patch mentioned above from glibc? If
reverting that works, we probably need to ask Arch folks to do the
same.

cheers,
--renato

8

I've looked at the version of libc I have and its 2.24:
$ /lib/libc.so.6
GNU C Library (GNU libc) stable release version 2.24, by Roland McGrath et al.

I then cloned the gcc 2.24 sources as of today and the code that
24e2b1cede1952d7d4411a3cafd25dd8593dab9f reverts is still there. I
also took a quick look at the Arch Linux glibc package
(Groups · Explore · GitLab)
and don't see any local patches.

Therefore I assume it hasn't been patched.

9

Right, I'm not expecting glibc itself to be patched by now, but if
this impacts Arch and Open Mandriva, then both should revert locally,
at least until it's fixed upstream.

I also didn't see any patch on the Arch and I'm not sure how to
contact the packager.

Can you reply on the bug you opened about the (slight) progress we're
making? Ask someone to try reverting the patch? I don't have a chroot
just for that and I don't want to mess up my environment. :slight_smile:

cheers,
--renato

10

I've updated the arch linux bug (FS#50385 : [clang] fsanitize=memory Segmenation fault) with
a patch for glibc in the arch linux packages that does fix the segment
fault for me.

11

https://reviews.llvm.org/D24736 would probably fix this

12

I hope so.