memory hog llvm-ld

Holger_Schurig1 · July 18, 2007, 11:44am

I want to share a little LLVM experiment.

I tried LLVM on one of my bigger Qt 3.x based projects. I used
llvm from SVN trunk (r39999) and SVN llvm-gcc-4.01 (r370) and
did compile every file with

  /usr/src/llvm/dist/bin/g++ -c -pipe -g \
  -Wall -Wextra -Wno-sign-compare \
  ... lots of -Dxxxx ... \
  --emit-llvm -I/usr/share/qt3/mkspecs/default \
  -I. -I.. -I../../../include/qt3 -I.obj/ \
  -o .obj/FILENAME.o FILENAME.cpp

so that I later had 211 .o files, all in the binary bytecode
format. I tried it that way because I thought that then the
possibilities for inter-process and inter-module optimizations
are probably the highest.

Note that I did specify "-g", but not any "-Ox" switches. That
made the size of all *.o files together being 143 MB.

Now, I tried to link this. I used this command line:

  /usr/src/llvm/dist/bin/llvm-ld -v -stats -native
  -o main -L/usr/share/qt3/lib -L/usr/X11R6/lib \
  -lcrypto -lusb -lutil -lqt-mt -lXext -lX11 -lm -lpthrea

used a tremendous amount of memory. My system swapped heavily.
After about a minute, I issued a "top" command and saw:

VIRT RES SHR S %CPU %MEM TIME+ COMMAND
454m 423m 904 R 27.8 84.2 1:06.51 llvm-ld

I then stopped llvm-ld, because I didn't know how my OOM memory
killer would behave

Holger_Schurig1 · July 18, 2007, 3:06pm

Seems like the memory-hog status is related to debug info, e.g. to -g. Because I later compiled the same program with RELEASE

settings, e.g.

$ /usr/src/llvm/dist/bin/g++ -c -pipe --emit-llvm \

-fvisibility=hidden -fvisibility-inlines-hidden \

-Wall -Wextra -Wno-sign-compare -Os \

… lots of -Dxxxx … \

-I/usr/share/qt3/mkspecs/default -I. -I… \

-I…/…/…/include/qt3 -I.obj/ \

-o .obj/FILENAME.o .obj/FILENAME.cpp

and linked the result like this:

$ /usr/src/llvm/dist/bin/llvm-ld \

-v -stats -native -verify-each -strip-all \

-o main \

… lots of *.o files … \

-L/usr/share/qt3/lib -L/usr/X11R6/lib \

-lcrypto -lusb -lutil -lqt-mt -lXext -lX11 -lm -lpthread

This time LLVM did not eat too much memory. I got a nice

executable.

$ size main

text data bss dec hex filename

1170074 74586 8456 1253116 131efc main

The same compiled with "g++ (GCC) 4.1.2 20061115 (prerelease)

(Debian 4.1.1-21)" and “GNU ld version 2.17 Debian GNU/Linux”

gave me:

$ size main

952074 8572 4528 965174 eba36 main

In the gcc case, I link with “g++”, with link options "-Wl,–gc-sections -Wl,-O2 -Wl,–as-needed

-Wl,–enable-new-dtags -Wl,-s"

Duncan_Sands · July 18, 2007, 3:38pm

This time LLVM did not eat too much memory. I got a nice
executable.

$ size main
text data bss dec hex filename
1170074 74586 8456 1253116 131efc main

The same compiled with "g++ (GCC) 4.1.2 20061115 (prerelease)
(Debian 4.1.1-21)" and "GNU ld version 2.17 Debian GNU/Linux"
gave me:

$ size main
952074 8572 4528 965174 eba36 main

In the gcc case, I link with "g++", with link options "-Wl,--gc-sections -Wl,-O2 -Wl,--as-needed
-Wl,--enable-new-dtags -Wl,-s"

You can also give -O options to llvm-ld, from -O0 to -O5.

Ciao,

Duncan.

Christopher_Lamb · July 18, 2007, 5:06pm

Perhaps this is related to the binutils bug discussed on the list recently (in certain versions of 2.17, I believe) that causes the native linker to behave very poorly when linking with debug symbols?

http://lists.cs.uiuc.edu/pipermail/llvmdev/2007-June/009391.html

Holger_Schurig1 · July 18, 2007, 7:48pm

> Seems like the memory-hog status is related to debug info, e.g.
> to -g.

Perhaps this is related to the binutils bug discussed on the list
recently (in certain versions of 2.17, I believe) that causes the
native linker to behave very poorly when linking with debug symbols?

I don't think so. The *.o files where in binary bitcode format, and
AFAIK llvm-ld reads them and runs various optimization stages on this.
I can hardly see how this can have anything to do with binutils. Before
the main.bc file got written, I terminated the run of llvm-ld.

Also, I'm not absolutely sure if it was because of "-g" or because I did
not specify any "-O" while compiling the c/c++ files into bitcode.

Reid_Spencer · July 19, 2007, 12:14am

> > Seems like the memory-hog status is related to debug info, e.g.
> > to -g.

> Perhaps this is related to the binutils bug discussed on the list
> recently (in certain versions of 2.17, I believe) that causes the
> native linker to behave very poorly when linking with debug symbols?

I don't think so. The *.o files where in binary bitcode format, and
AFAIK llvm-ld reads them and runs various optimization stages on this.
I can hardly see how this can have anything to do with binutils. Before
the main.bc file got written, I terminated the run of llvm-ld.

That's correct, llvm-ld does nothing with binutils. If its linking
natively, it invokes gcc in a separate process which does use binutils,
but the memory wouldn't be attributed to llvm-ld in that case.

Also, I'm not absolutely sure if it was because of "-g" or because I did
not specify any "-O" while compiling the c/c++ files into bitcode.

Either way the output from llvm-gcc is much more verbose than a run
without -g or with -Ox

Reid.

Duncan_Sands · July 19, 2007, 7:43am

Hi Holger,

Note that I did specify "-g", but not any "-Ox" switches. That
made the size of all *.o files together being 143 MB.

LLVM represents debug info as explicit calls to intrinsics.
This approach has many advantages, but a possible disadvantage
is that it can significantly increase the size of the bitcode.
I don't know if that explains your observations. I'm curious
to know how gcc stores debug info...

Ciao,

Duncan.

Holger_Schurig1 · July 19, 2007, 10:57am

LLVM represents debug info as explicit calls to intrinsics.
This approach has many advantages, but a possible disadvantage
is that it can significantly increase the size of the bitcode.
I don't know if that explains your observations. I'm curious
to know how gcc stores debug info...

GCC stores debug info in ELF sections of the .o file, e.g.:

$ objdump -h wtd.o

wtd.o: file format elf32-i386

Sections:
Idx Name Size VMA LMA File off Algn
  0 .text 00000ef4 00000000 00000000 00000034 2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data 00000000 00000000 00000000 00000f28 2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss 00000000 00000000 00000000 00000f28 2**2
                  ALLOC
  3 .debug_abbrev 0000017e 00000000 00000000 00000f28 2**0
                  CONTENTS, READONLY, DEBUGGING
  4 .debug_info 00001052 00000000 00000000 000010a6 2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  5 .debug_line 00000229 00000000 00000000 000020f8 2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
  6 .rodata 00000161 00000000 00000000 00002324 2**2
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  7 .debug_frame 00000140 00000000 00000000 00002488 2**2
                  CONTENTS, RELOC, READONLY, DEBUGGING
  8 .debug_loc 00000210 00000000 00000000 000025c8 2**0
                  CONTENTS, READONLY, DEBUGGING
  9 .debug_pubnames 00000089 00000000 00000000 000027d8 2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
10 .debug_aranges 00000020 00000000 00000000 00002861 2**0
                  CONTENTS, RELOC, READONLY, DEBUGGING
11 .debug_str 0000005b 00000000 00000000 00002881 2**0
                  CONTENTS, READONLY, DEBUGGING
12 .comment 0000003a 00000000 00000000 000028dc 2**0
                  CONTENTS, READONLY
13 .note.GNU-stack 00000000 00000000 00000000 00002916 2**0
                  CONTENTS, READONLY

When I compile this file without debug info, I get this:

$ objdump -h wtd.o

wtd.o: file format elf32-i386

Sections:
Idx Name Size VMA LMA File off Algn
  0 .text 00000871 00000000 00000000 00000034 2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data 00000000 00000000 00000000 000008a8 2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss 00000000 00000000 00000000 000008a8 2**2
                  ALLOC
  3 .rodata.str1.1 00000157 00000000 00000000 000008a8 2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .comment 0000003a 00000000 00000000 000009ff 2**0
                  CONTENTS, READONLY
  5 .note.GNU-stack 00000000 00000000 00000000 00000a39 2**0
                  CONTENTS, READONLY

There are various formats available, e.g. Dwarf and others. I
think this is all handled via libbfd, or you can get more info
by staring at the gdb source.

Duncan_Sands · July 19, 2007, 11:31am

Hi Holger,

Daniel_Berlin1 · July 19, 2007, 2:13pm

It doesn't.

It is generated after optimizations are performed (though we do track
some things through optimizations).