Stack smashing

Someone is trying to work on HLVM with me but they're hitting a problem that
we have not been able to resolve. Specifically, GCC seems to be performing
some kind of sanity check for "stack smashing" and is calling abort because
it is unhappy with something that the code is doing. However, I am not sure
what and cannot reproduce the problem.

The stack trace they have given me is:

*(gdb) run
Starting program: /home/alp/ocaml/hlvm/hlvm
[Thread debugging using libthread_db enabled]
*** stack smashing detected ***: /home/alp/ocaml/hlvm/hlvm terminated
[New Thread 0xb7dc56c0 (LWP 25458)]
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x48)[0xb7ec16d8]
/lib/tls/i686/cmov/libc.so.6(__fortify_fail+0x0)[0xb7ec1690]
/home/alp/ocaml/hlvm/hlvm(_ZN4llvm3JIT11runFunctionEPNS_8FunctionERKSt6vectorINS_12GenericValueESaIS4_EE+0xc53)
[0x852f7d3]
======= Memory map: ========
08048000-087ce000 r-xp 00000000 08:05 4588454 /home/alp/ocaml/hlvm/hlvm
087ce000-087cf000 r--p 00785000 08:05 4588454 /home/alp/ocaml/hlvm/hlvm
087cf000-087d4000 rw-p 00786000 08:05 4588454 /home/alp/ocaml/hlvm/hlvm
087d4000-087db000 rw-p 087d4000 00:00 0
09449000-0948b000 rw-p 09449000 00:00 0 [heap]
b6d40000-b7d40000 rwxp b6d40000 00:00 0
b7d40000-b7dc7000 rw-p b7d40000 00:00 0
b7dc7000-b7f1f000 r-xp 00000000 08:05 2138182 /lib/tls/i686/cmov/
libc-2.8.90.so
b7f1f000-b7f21000 r--p 00158000 08:05 2138182 /lib/tls/i686/cmov/
libc-2.8.90.so
b7f21000-b7f22000 rw-p 0015a000 08:05 2138182 /lib/tls/i686/cmov/
libc-2.8.90.so
b7f22000-b7f25000 rw-p b7f22000 00:00 0
b7f25000-b7f32000 r-xp 00000000 08:05 2122025 /lib/libgcc_s.so.1
b7f32000-b7f33000 r--p 0000c000 08:05 2122025 /lib/libgcc_s.so.1
b7f33000-b7f34000 rw-p 0000d000 08:05 2122025 /lib/libgcc_s.so.1
b7f34000-b7f61000 r-xp 00000000 08:05 2121805 /lib/libncurses.so.5.6
b7f61000-b7f64000 rw-p 0002c000 08:05 2121805 /lib/libncurses.so.5.6
b7f64000-b7f66000 r-xp 00000000 08:05 4358492
/usr/lib/libsigsegv.so.0.0.0
b7f66000-b7f68000 rw-p 00001000 08:05 4358492
/usr/lib/libsigsegv.so.0.0.0
b7f68000-b804b000 r-xp 00000000 08:05 5177913
/usr/lib/libstdc++.so.6.0.10
b804b000-b804f000 r--p 000e3000 08:05 5177913
/usr/lib/libstdc++.so.6.0.10
b804f000-b8050000 rw-p 000e7000 08:05 5177913
/usr/lib/libstdc++.so.6.0.10
b8050000-b8056000 rw-p b8050000 00:00 0
b8056000-b807a000 r-xp 00000000 08:05 2138186 /lib/tls/i686/cmov/
libm-2.8.90.so
b807a000-b807b000 r--p 00023000 08:05 2138186 /lib/tls/i686/cmov/
libm-2.8.90.so
b807b000-b807c000 rw-p 00024000 08:05 2138186 /lib/tls/i686/cmov/
libm-2.8.90.so
b807c000-b807d000 rw-p b807c000 00:00 0
b807d000-b807f000 r-xp 00000000 08:05 2138185 /lib/tls/i686/cmov/
libdl-2.8.90.so
b807f000-b8080000 r--p 00001000 08:05 2138185 /lib/tls/i686/cmov/
libdl-2.8.90.so
b8080000-b8081000 rw-p 00002000 08:05 2138185 /lib/tls/i686/cmov/
libdl-2.8.90.so
b8081000-b8096000 r-xp 00000000 08:05 2138198 /lib/tls/i686/cmov/
libpthread-2.8.90.so
b8096000-b8097000 r--p 00014000 08:05 2138198 /lib/tls/i686/cmov/
libpthread-2.8.90.so
b8097000-b8098000 rw-p 00015000 08:05 2138198 /lib/tls/i686/cmov/
libpthread-2.8.90.so
b8098000-b809a000 rw-p b8098000 00:00 0
b80b1000-b80b3000 rw-p b80b1000 00:00 0
b80b3000-b80cd000 r-xp 00000000 08:05 2122026 /lib/ld-2.8.90.so
b80cd000-b80ce000 r-xp b80cd000 00:00 0 [vdso]
b80ce000-b80cf000 r--p 0001a000 08:05 2122026 /lib/ld-2.8.90.so
b80cf000-b80d0000 rw-p 0001b000 08:05 2122026 /lib/ld-2.8.90.so
bfeba000-bfecf000 rw-p bffeb000 00:00 0 [stack]

Program received signal SIGABRT, Aborted.
[Switching to Thread 0xb7dc56c0 (LWP 25458)]
0xb80cd430 in __kernel_vsyscall ()
(gdb) bt
#0 0xb80cd430 in __kernel_vsyscall ()
#1 0xb7df28a0 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7df4268 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb7e3016d in ?? () from /lib/tls/i686/cmov/libc.so.6
#4 0xb7ec16d8 in __fortify_fail () from /lib/tls/i686/cmov/libc.so.6
#5 0xb7ec1690 in __stack_chk_fail () from /lib/tls/i686/cmov/libc.so.6
#6 0x0852f7d3 in llvm::JIT::runFunction ()

From a cursory glance, it looks like something is messing with the

stack canarys. Probably a stack buffer overflow.

In case it is relevant, HLVM uses libsigsegv to detect stack overflows and
that stack handler is initialized in my JITted code which LLVM's runFunction
should be calling.

Could libsigsegv be conflicing with the stack smashing code?

If it changes known values on the stack - yes.

Basically it all works by placing a sentinel value on the stack initialized with a
random number that's then checked at the end of the function. If that has been
changed it calls abort(). IIRC the feature is turned on by default on modern
versions of linux. I do not believe it is yet turned on by default for darwin.

-eric

It's not turned on by default for Leopard. If you don't want these
stack protectors (which is sounds like in this case), just use
-fno-stack-protector. At least on the particular files you know that
libsigsegv is messing with...

-bw