Slow debugger starts of LLVM tools

I'm wondering if anyone else is running into this problem.

When I do a Debug build and attempt to run a tool (opt, llvm-dis,
whatever) in gdb, the program takes a LOOOOONG time to begin executing.
Loading the executable into gdb is not the problem. I type "run" and
the debugger sits there for many seconds and then things start going.
gdb emits no messages indicating what it is doing but it's basically
using 100% CPU.

I'm wondering if this is an artifact of the gdb I'm using (buggy?) or
something inherent to the way LLVM tools are built. I see the same slow
operation across different versions of gdb and I do not see the same
slow operation with other executables.

                         -David

I'm wondering if anyone else is running into this problem.

When I do a Debug build and attempt to run a tool (opt, llvm-dis,
whatever) in gdb, the program takes a LOOOOONG time to begin executing.
Loading the executable into gdb is not the problem. I type "run" and
the debugger sits there for many seconds and then things start going.
gdb emits no messages indicating what it is doing but it's basically
using 100% CPU.

I do see such perf issues, although i *think* for me they happen
when initially loading the executable in gdb, and the 'run' itself works fast.

I'm wondering if this is an artifact of the gdb I'm using (buggy?) or
something inherent to the way LLVM tools are built. I see the same slow
operation across different versions of gdb and I do not see the same
slow operation with other executables.

It probably has the same source as the Release vs Debug filesize -
debug info for LLVM is *big*.
(It would be interesting to know if there is a solution, other than
using lldb of course.)

                         -David

Roman.

GDB likes to load all symbols from shared libraries up front. And on x86_64 your main executable is really just another shared library.

So yes, it’s slow. The sloth is proportional to the size of the object(s) being debugged. It’s not just LLVM, it’s any large (i.e. 100+MB to 1+GB) executable+library footprint.

David Jones via llvm-dev <llvm-dev@lists.llvm.org> writes:

GDB likes to load all symbols from shared libraries up front. And on
x86_64 your main executable is really just another shared library.

Yes, but does gdb reload everything on each execution? Every time I
execute "run" I see the same slow behavior. Loading the symbols for
small tools like llvm-rc takes hardly any time at all (i.e. "file
llvm-rc" is almost instantaneous). But executing it causes gdb to chew
up CPU cycles for a while. Every time.

                         -David

I don't know about the issues with running being slow, but using a GDB index greatly speeds up initial symbol loading. Compile with `-ggnu-pubnames` and link with `-Xlinker --gdb-index` and you should get significantly faster symbol load times. (I'd be curious to see if it helps with the issues with run being slow as well.)

    David Jones via llvm-dev <llvm-dev@lists.llvm.org> writes:
    
    > GDB likes to load all symbols from shared libraries up front. And on
    > x86_64 your main executable is really just another shared library.
    
    Yes, but does gdb reload everything on each execution? Every time I
    execute "run" I see the same slow behavior. Loading the symbols for
    small tools like llvm-rc takes hardly any time at all (i.e. "file
    llvm-rc" is almost instantaneous). But executing it causes gdb to chew
    up CPU cycles for a while. Every time.
    
                             -David

This is admittedly a longshot, but Can you check whether you experience unusually long run times with LLDB? If there’s something strange about tge binaries we generate, maybe lldb will exhibit some strangeness too and we can more easily profile it.

I’ve ran into the same issue before. I tend to see it as a price we pay for static linking, since it brings all the symbols debugger has to read at the start time. Quick check on launching a tool (fort) in GDB (I usually use it) and LLDB shows that LLDB starts much faster, but takes time to set breakpoints and launch the application, so I think they are roughly on the same page in terms of delays.

-Petr

Yep, as others have mentioned - using a linker-generated gdb-index is super helpful (10s for my machine to start debugging an unoptimized clang, set a breakpoint at llvm::StringRef::StringRef and run until that breakpoint is hit, as opposed to 1m26s without an index)

That’s also with/without split DWARF too, but I suspect most of the benefit is in the index there. Split DWARF might save you some linking time and /maybe/ some debugger performance.

I was under the impression that a generated gdb-index only helps when
loading the file ('file foo' in gdb). Does it help execution startup
time as well?

I'll give it a try and see if it helps. Thanks all!

                         -David

David Blaikie via llvm-dev <llvm-dev@lists.llvm.org> writes:

Ah, yeah, I haven’t observed particularly long program startup time. Could you time it & specify exxactly which commands (both the command you used to start gdb, and the commands you executed in gdb) you executed/which ones took a long time/roughly how long they took, etc?