ld crashes with invalid pointer on Ubuntu 11.04 x86 when linking libgcc_s.so

This shouldn't be considered our bug (should it?). I'm going to
report it to Ubuntu.

I will restart my build, but does anyone else see this on Ubuntu
11.04? Note that I have a 32-bit machine with 2 GB of RAM and no
swap.

$ ld --version
GNU ld (GNU Binutils for Ubuntu) 2.21.0.20110327
Copyright 2011 Free Software Foundation, Inc.

$ uname -a
Linux frylock 2.6.38-11-generic #50-Ubuntu SMP Mon Sep 12 21:18:14 UTC
2011 i686 i686 i386 GNU/Linux

I have an Early 2006 MacBook Pro, Model Identifier MacBookPro1,1, with
a 1.83 GHz Core Duo (NOT Core 2 Duo), with 2GB of RAM.

I didn't set up a swap partition because there is also the EFI boot
partition, Mac OS X Snow Leopard and Windows XP. More Than Four
Partitions Makes The Baby Bootcamp cry. I guess I could swap into a
regular file but I just haven't dealt with setting that up yet; for
the most part 2 GB is enough.

So maybe ld ran out of memory but didn't check its allocations carefully enough.

You can see from my build log that it was executing
projects/llvm-gcc-4.2/./gcc/xgcc with about a bazillion shared
libraries on the command line:

/home/mike/LLVM/r140981/build/projects/llvm-gcc-4.2/./gcc/xgcc
-B/home/mike/LLVM/r140981/build/projects/llvm-gcc-4.2/./gcc/
-B/home/mike/LLVM/r140981/i686-pc-linux-gnu/bin/
-B/home/mike/LLVM/r140981/i686-pc-linux-gnu/lib/ -isystem
/home/mike/LLVM/r140981/i686-pc-linux-gnu/include -isystem
/home/mike/LLVM/r140981/i686-pc-linux-gnu/sys-include -O2 -O2 -g -O2
-DIN_GCC -W -Wall -Wwrite-strings -Wstrict-prototypes
-Wmissing-prototypes -Wold-style-definition -isystem ./include -fPIC
-g -DHAVE_GTHR_DEFAULT -DIN_LIBGCC2 -D__GCC_FLOAT_NOT_NEEDED -shared
-nodefaultlibs -Wl,--soname=libgcc_s.so.1
-Wl,--version-script=libgcc/./libgcc.map -o ./libgcc_s.so.1.tmp
libgcc/./_muldi3_s.o libgcc/./_negdi2_s.o libgcc/./_lshrdi3_s.o
libgcc/./_ashldi3_s.o libgcc/./_ashrdi3_s.o libgcc/./_cmpdi2_s.o
libgcc/./_ucmpdi2_s.o libgcc/./_clear_cache_s.o
libgcc/./_enable_execute_stack_s.o libgcc/./_trampoline_s.o
libgcc/./__main_s.o libgcc/./_absvsi2_s.o libgcc/./_absvdi2_s.o
libgcc/./_addvsi3_s.o libgcc/./_addvdi3_s.o libgcc/./_subvsi3_s.o
libgcc/./_subvdi3_s.o libgcc/./_mulvsi3_s.o libgcc/./_mulvdi3_s.o
libgcc/./_negvsi2_s.o libgcc/./_negvdi2_s.o libgcc/./_ctors_s.o
libgcc/./_ffssi2_s.o libgcc/./_ffsdi2_s.o libgcc/./_clz_s.o
libgcc/./_clzsi2_s.o libgcc/./_clzdi2_s.o libgcc/./_ctzsi2_s.o
libgcc/./_ctzdi2_s.o libgcc/./_popcount_tab_s.o
libgcc/./_popcountsi2_s.o libgcc/./_popcountdi2_s.o
libgcc/./_paritysi2_s.o libgcc/./_paritydi2_s.o libgcc/./_powisf2_s.o
libgcc/./_powidf2_s.o libgcc/./_powixf2_s.o libgcc/./_powitf2_s.o
libgcc/./_mulsc3_s.o libgcc/./_muldc3_s.o libgcc/./_mulxc3_s.o
libgcc/./_multc3_s.o libgcc/./_divsc3_s.o libgcc/./_divdc3_s.o
libgcc/./_divxc3_s.o libgcc/./_divtc3_s.o libgcc/./_bswapsi2_s.o
libgcc/./_bswapdi2_s.o libgcc/./_fixunssfsi_s.o
libgcc/./_fixunsdfsi_s.o libgcc/./_fixunsxfsi_s.o
libgcc/./_fixsfdi_s.o libgcc/./_fixunssfdi_s.o libgcc/./_floatdisf_s.o
libgcc/./_floatundisf_s.o libgcc/./_fixdfdi_s.o
libgcc/./_fixunsdfdi_s.o libgcc/./_floatdidf_s.o
libgcc/./_floatundidf_s.o libgcc/./_fixxfdi_s.o
libgcc/./_fixunsxfdi_s.o libgcc/./_floatdixf_s.o
libgcc/./_floatundixf_s.o libgcc/./_fixtfdi_s.o
libgcc/./_fixunstfdi_s.o libgcc/./_floatditf_s.o
libgcc/./_floatunditf_s.o libgcc/./_divdi3_s.o libgcc/./_moddi3_s.o
libgcc/./_udivdi3_s.o libgcc/./_umoddi3_s.o libgcc/./_udiv_w_sdiv_s.o
libgcc/./_udivmoddi4_s.o libgcc/./unwind-dw2_s.o
libgcc/./unwind-dw2-fde-glibc_s.o libgcc/./unwind-sjlj_s.o
libgcc/./gthr-gnat_s.o libgcc/./unwind-c_s.o -lc && rm -f
./libgcc_s.so && if [ -f ./libgcc_s.so.1 ]; then mv -f ./libgcc_s.so.1
./libgcc_s.so.1.backup; else true; fi && mv ./libgcc_s.so.1.tmp
./libgcc_s.so.1 && ln -s libgcc_s.so.1 ./libgcc_s.so
*** glibc detected *** /usr/bin/ld: munmap_chunk(): invalid pointer:
0x08d3a6e4 ***
======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.6(+0x6b961)[0x4019e961]
/lib/i386-linux-gnu/libc.so.6(+0x6c10e)[0x4019f10e]
/usr/lib/libbfd-2.21.0-system.20110327.so(_bfd_elf_discard_section_eh_frame+0x50a)[0x400afeea]
/usr/lib/libbfd-2.21.0-system.20110327.so(bfd_elf_discard_info+0x2f4)[0x400a0994]
/usr/bin/ld[0x806a763]
/usr/bin/ld[0x805b5d0]
/usr/bin/ld[0x806078e]
/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x40149e37]
/usr/bin/ld[0x804b0d1]
======= Memory map: ========
08048000-080cd000 r-xp 00000000 08:03 143954 /usr/bin/ld.bfd
080cd000-080ce000 r--p 00084000 08:03 143954 /usr/bin/ld.bfd
080ce000-080cf000 rw-p 00085000 08:03 143954 /usr/bin/ld.bfd
080cf000-080d0000 rw-p 00000000 00:00 0
0896d000-08db4000 rw-p 00000000 00:00 0 [heap]
40000000-4001c000 r-xp 00000000 08:03 1048975 /lib/i386-linux-gnu/ld-2.13.so
4001c000-4001d000 r--p 0001b000 08:03 1048975 /lib/i386-linux-gnu/ld-2.13.so
4001d000-4001e000 rw-p 0001c000 08:03 1048975 /lib/i386-linux-gnu/ld-2.13.so
4001e000-4001f000 r-xp 00000000 00:00 0 [vdso]
4001f000-4002d000 rw-p 00000000 00:00 0
4003a000-4010a000 r-xp 00000000 08:03 131893
/usr/lib/libbfd-2.21.0-system.20110327.so
4010a000-40113000 r--p 000d0000 08:03 131893
/usr/lib/libbfd-2.21.0-system.20110327.so
40113000-40115000 rw-p 000d9000 08:03 131893
/usr/lib/libbfd-2.21.0-system.20110327.so
40115000-4011a000 rw-p 00000000 00:00 0
4011a000-4012d000 r-xp 00000000 08:03 1048635
/lib/i386-linux-gnu/libz.so.1.2.3.4
4012d000-4012e000 r--p 00012000 08:03 1048635
/lib/i386-linux-gnu/libz.so.1.2.3.4
4012e000-4012f000 rw-p 00013000 08:03 1048635
/lib/i386-linux-gnu/libz.so.1.2.3.4
4012f000-40131000 r-xp 00000000 08:03 1048972
/lib/i386-linux-gnu/libdl-2.13.so
40131000-40132000 r--p 00001000 08:03 1048972
/lib/i386-linux-gnu/libdl-2.13.so
40132000-40133000 rw-p 00002000 08:03 1048972
/lib/i386-linux-gnu/libdl-2.13.so
40133000-4028d000 r-xp 00000000 08:03 1048755
/lib/i386-linux-gnu/libc-2.13.so
4028d000-4028e000 ---p 0015a000 08:03 1048755
/lib/i386-linux-gnu/libc-2.13.so
4028e000-40290000 r--p 0015a000 08:03 1048755
/lib/i386-linux-gnu/libc-2.13.so
40290000-40291000 rw-p 0015c000 08:03 1048755
/lib/i386-linux-gnu/libc-2.13.so
40291000-40295000 rw-p 00000000 00:00 0
402ae000-402c8000 r-xp 00000000 08:03 1048781
/lib/i386-linux-gnu/libgcc_s.so.1
402c8000-402c9000 r--p 00019000 08:03 1048781
/lib/i386-linux-gnu/libgcc_s.so.1
402c9000-402ca000 rw-p 0001a000 08:03 1048781
/lib/i386-linux-gnu/libgcc_s.so.1
bfae3000-bfb07000 rw-p 00000000 00:00 0 [stack]
collect2: ld terminated with signal 6 [Aborted]
make[4]: *** [libgcc_s.so] Error 1

My shell history gives my configure and make command lines:

2011 cd
2012 cd LLVM/r140981/build/projects/llvm-gcc-4.2/
2013 fg
2014 ../../../../SVN/llvm/projects/llvm-gcc-4.2/configure
--prefix=/home/mike/LLVM/r140981 --target=i686-pc-linux-gnu
--with-tune=generic --with-arch=i686
--enable-languages=c,c++,objc,obj-c++
LLVM_VERSION_INFO=3.0_svn_r140981
--enable-llvm=/home/mike/LLVM/r140981 --enable-debug-runtime
--enable-jit
2015 ../../../../SVN/llvm/projects/llvm-gcc-4.2/configure
--prefix=/home/mike/LLVM/r140981 --target=i686-pc-linux-gnu
--with-tune=generic --with-arch=i686
--enable-languages=c,c++,objc,obj-c++ LLVM_VERSION_INFO=3.0
--enable-llvm=/home/mike/LLVM/r140981 --enable-debug-runtime
--enable-jit
2016 ../../../../SVN/llvm/projects/llvm-gcc-4.2/configure
--prefix=/home/mike/LLVM/r140981 --target=i686-pc-linux-gnu
--with-tune=generic --with-arch=i686
--enable-languages=c,c++,objc,obj-c++
--enable-llvm=/home/mike/LLVM/r140981 --enable-debug-runtime
--enable-jit
2017 make -j3 LLVM_VERSION_INFO=3.0+svn_r140981
2018 make -j3 LLVM_VERSION_INFO=3.0+svn_r140981 clean
2019 make -j3 LLVM_VERSION_INFO=3.0+svn_r140981 bootstrap

My object (build) directory is ~/LLVM/r140981, because my latest "svn
update" got revision 140981, and my configure --prefix is
~/LLVM/r140981, so when I do a "make install" it puts my executables
in ~/LLVM/r140981/bin. I put a one-line shell script in my LLVM
directory to prepend ~/LLVM/r140981/bin to my $PATH.

I downloaded the source package for Ubuntu 11.04's ld - it's part of
binutils. It's building right now with "-g" on the compile command
line, and with --prefix set to install in a subdir of my ~/LLVM
directory.

I've already tried enabling core dumps:

   $ ulimit -c unlimited

... but with the release ld that's included with Ubuntu I don't get
symbols. Building with symbols will enable to debug the core file.

My guess is that we're linking so many shared libraries that ld
overflows a buffer or some such.

I don't get the crash at all with the ld I built myself from the
Ubuntu 11.04 source deb. Instead I get a complaint that ld cannot
find -lc; I believe I had the --prefix path wrong when I built the
release llvm at the start, so that later "make bootstrap" in
llvm-gcc-4.2 looks in the wrong place.

I didn't start my build with a binary distro of llvm-gcc. I expect it
would have been easier if I had but I wanted to try bootstrapping the
source from scratch.

I'm not sure, but a possibility is that either ld or a library it
depends on has badly generated code. I built with debugging on, which
may have altered the generated i386 code.

I also built on a 32-bit Core Duo; I expect that whoever built the
optimized binaries for Ubuntu did so on an x86_64 box with a compiler
that produces i386 binaries, a sort of cross-compiler. We all know
that cross-compilers are cans of worms!

/usr/bin/ld is a symbolic link to /usr/bin/ld.bfd. Here is how I
rebuilt it myself:

First we need to know what Ubuntu package ld.bfd is part of:

$ sudo apt-get install apt-file

Apt-file needs to download some databases:

$ sudo apt-file update
$ apt-file search ld.bfd

Apt-file now says that ld.bfd is in binutils.

$ mkdir ~/DebSrc
$ cd ~/DebSrc
$ sudo apt-get source binutils

On Debian one would now use dpkg-source to unroll the source archive,
but "apt-get source" does this on its own on Ubuntu.

$ cd binutils-2.21.0.20110327

We might need some special tools, header files or development
libraries to do the build:

$ sudo apt-get build-dep binutils
$ sudo apt-get install build-essentials fakeroot dpkg-dev

Examination of the configure script seems to indicate we want to set
$CFLAGS_FOR_BUILD to "-g" rather than just $CFLAGS:

$ export CFLAGS_FOR_BUILD=-g

On Debian, $DEB_BUILD_OPTIONS will be passed on to configure, but it
didn't work for me on Ubuntu. That is, I tried:

$ export DEB_BUILD_OPTIONS=--prefix=/home/mike/binutils-2.21.0.20110327

... because I didn't want to clobber the official binutils, lest I
break everything real bad. But when I installed the binary debs, all
the executables and such went into /usr, just as if I made an official
build. I don't know what I might have done wrong.

Now build the binaries:

$ fakeroot debian/rules binary

The binary deb archives are placed in the parent directory to where
you are now (~/DebSrc):

$ cd ..
$ ls -l *.deb

(The listing shows several debs that were just created.)

Now install them. As I said I didn't want to install these over the
official packages, but somehow screwed up. Even so, I think I got a
good build so it's OK

$ sudo dpkg -i *.deb

I made a one-line - one VERY LONG line - script with the xgcc command
line that links libgcc_s.so.1. With the release /usr/bin/ld.bfd, it
dumps core, but with mine it says:

/usr/bin/ld: cannot find -lc

This seems to be because I didn't put my initial build of llvm where
the bootstrap llvm-gcc wants to find that. Once I make sure I've got
my configure command lines set up right and can build, I'll post a
followup to this threa.

Signal 6 is SIGABRT so you may want to find out what’s calling abort inside ld. An abort usually means that the linker has gotten into an inconsistent or unexpected state.

-eric

The abort is actually called from some libc subroutine that's a couple
levels below ld's code.

I'm close to having it build, I think.

It looks like the crash was caused by having the release 11.04 ld not
be able to find -lc. It's in /lib/i386-linux-gnu on Ubuntu rather
than just in /lib.

After I get LLVM built all the way I'll try putting the release
binutils back to see whether I can still reproduce the crash. It
might be that my homebuild ld's complaint that it can't find -lc
causes the release ld to abort.