Breakpoints from source-before-file are not working

Subprojects LLDB
#1

Hi list,

I have a file lldb.txt with LLDB commands:

target create PATH/TO/BINARY
br s -a 0x7fff5fc01031
process launch --stop-at-entry

So I can start debugging a program with lldb -S lldb.txt. After launch the debugger stops at 0x7fff5fc01000 (under Mac OS X 10.10) in dyld. As you can see I want to stop at breakpoint 0x7fff5fc01031 in dyld.

The breakpoint is unresolved but locations = 1.

[lldb]> breakpoint list
Current breakpoints:
1: address = 0x00007fff5fc01031, locations = 1
1.1: address = 0x00007fff5fc01031, unresolved, hit count = 0

When I add the breakpoint manually then it’s resolved.

[lldb]> br s -a 0x7fff5fc01031
Breakpoint 2: where = dyld`_dyld_start + 49, address = 0x00007fff5fc01031
[lldb]> breakpoint list
Current breakpoints:
1: address = 0x00007fff5fc01031, locations = 1
1.1: address = 0x00007fff5fc01031, unresolved, hit count = 0

2: address = 0x00007fff5fc01031, locations = 1, resolved = 1, hit count = 0
2.1: where = dyld`_dyld_start + 49, address = 0x00007fff5fc01031, resolved, hit count = 0

How can I add a breakpoint in my lldb.txt file which will be automatically resolved?

I don’t know if this is a bug or if I’m doing it wrong with breakpoints. I already checked out the newest version (lldb-330.99.0) from the LLDB Git repository and compiled it with xcodebuild. But it’s still unresolved.

Best regards,
Christian

#2

When I add a breakpoint like this then before I run the program it is not resolved, but then when I run it does get resolved and hit. For instance:

> cat /tmp/address-bkpt.lldb
break set -a 0x00007fff9223f050
> lldb Sketch.app/ -S /tmp/address-bkpt.lldb
(lldb) command source -s 1 '/tmp/address-bkpt.lldb'
(lldb) target create "Sketch.app"
Current executable set to 'Sketch.app' (x86_64).
(lldb) break list
Current breakpoints:
1: address = 0x00007fff9223f050, locations = 1
  1.1: address = 0x00007fff9223f050, unresolved, hit count = 0

(lldb) run
Process 62885 launched: 'Sketch.app/Contents/MacOS/Sketch' (x86_64)
Process 62885 stopped
* thread #1: tid = 0x2447b6, function: objc_retain , stop reason = breakpoint 1.1
    frame #0: 0x00007fff9223f050 libobjc.A.dylib`objc_retain
-> 0x7fff9223f050 <objc_retain>: xorl %eax, %eax
    0x7fff9223f052 <objc_retain+2>: testq %rdi, %rdi
    0x7fff9223f055 <objc_retain+5>: je 0x7fff9223f060 ; objc_retain + 16
    0x7fff9223f057 <objc_retain+7>: testb $0x1, %dil

Address breakpoints are funny before you run, since libraries haven't gotten their correct load addresses, and in fact quite often there's either nothing actually in the address where they WILL load, or many things, because most libraries on OS X are zero based.

So I wouldn't worry too much about what it says before you run. If it isn't hitting the breakpoint once you actually run, however, that would be worth look into.

Jim

#3

It isn't hitting the breakpoint after I "run". When the program is
running and I'm at the address of the breakpoint the breakpoint is still
marked as "unresolved". I wouldn't have asked if it would work correctly.

As you can see below I manually went to the address of the breakpoint
after "process launch --stop-at-entry" and the breakpoint is still not
resolved.

[lldb]> ni
Process 5666 stopped
* thread #1: tid = 0x45407, 0x00007fff5fc01031 dyld`_dyld_start + 49,
stop reason = instruction step over
    frame #0: 0x00007fff5fc01031 dyld`_dyld_start + 49
-> 0x7fff5fc01031 <_dyld_start+49>: callq 0x7fff5fc01076 ;
dyldbootstrap::start(macho_header const*, int, char const**, long,
macho_header const*, unsigned long*)
   0x7fff5fc01036 <_dyld_start+54>: movq -0x8(%rbp), %rdi
   0x7fff5fc0103a <_dyld_start+58>: cmpq $0x0, %rdi
   0x7fff5fc0103e <_dyld_start+62>: jne 0x7fff5fc01050 ;
_dyld_start + 80
[lldb]> br li
Current breakpoints:
1: address = 0x00007fff5fc01031, locations = 1
  1.1: address = 0x00007fff5fc01031, unresolved, hit count = 0

After I stepped over the breakpoint it looks like this:

[lldb]> ni
Process 5666 stopped
* thread #1: tid = 0x45407, 0x00007fff5fc01036 dyld`_dyld_start + 54,
queue = 'com.apple.main-thread', stop reason = instruction step over
    frame #0: 0x00007fff5fc01036 dyld`_dyld_start + 54
-> 0x7fff5fc01036 <_dyld_start+54>: movq -0x8(%rbp), %rdi
   0x7fff5fc0103a <_dyld_start+58>: cmpq $0x0, %rdi
   0x7fff5fc0103e <_dyld_start+62>: jne 0x7fff5fc01050 ;
_dyld_start + 80
   0x7fff5fc01040 <_dyld_start+64>: movq %rbp, %rsp
[lldb]> br li
Current breakpoints:
1: address = 0x00007fff5fc01031, locations = 1, resolved = 1, hit count = 0
  1.1: address = 0x00007fff5fc01031, resolved, hit count = 0

And I tried the same procedure as you with only one line in my txt file
and after I hit "run" on a simple "hello world" C program the program
exited with status 0 without hitting the breakpoint at 0x7fff5fc01031.

Christian

#4

I didn't notice you were stopping early in dyld_start. lldb doesn't try to resolve any breakpoints that haven't currently been resolved that early on, since it knows the world is going to change out from under it, so for 99.999% of all breakpoints that work will be wasted. If you set breakpoints that are going to take after the first shared library loads are completed (which is what the example I showed does) then they will take successfully. Actually, your address breakpoint in dyld_start took as well, it just did so after the first set of shared library loads, i.e. after the code you wanted to break on was run.

Feel free to file a bug about this. Maybe we can do something like: if you specify the shared library to be the dynamic loader (through the -s option) we'll set that breakpoint when we stop on exec.

Jim

#5

So with the current version of LLDB the source-before-file is worthless
for debugging a shared library? So this is acctually not a bug?

Is there a workaround to debug a shared library with the current version
of LLDB? Setting all breakpoints manually for each start is too much
work. If you must set the breakpoints manually the source-before-file is
worthless in my eyes. And even if I don't restart the LLDB itself and
only the program I would like to debug the breakpoints are again
unresolved after re-start.

What do you mean with

since it knows the world is going to change out from under it

? The addresses of a loaded library doesn't change from run to run.

Christian

#6

So with the current version of LLDB the source-before-file is worthless
for debugging a shared library? So this is acctually not a bug?

Two things here. First of all, up till very recently lldb didn't support setting breakpoints before you have a target. That was a known bug, which I fixed a short while ago (but there hasn't been a new release of the tools from Apple since that fix went in.) Sorry I didn't mention that, I assumed from the fact that you were building TOT lldb to test this that you knew that. That this didn't work up till now was clearly a bug, that's why I fixed it.

With TOT lldb, however this does work, as I showed in the example I sent. You sent another example, for which thanks, BTW. I explained that the reason that example didn't work was you were trying to set a breakpoint in code that ran before ANY shared libraries are mapped into the program. So while your example showed a small set of code for which "source-before-file" breakpoints don't work, that code is not in shared libraries, it is in the loader, and at such an early stage that nothing of any importance has happened in the program yet.

AND I said that was a bug, please file it. So this comment seems totally off base.

Is there a workaround to debug a shared library with the current version
of LLDB? Setting all breakpoints manually for each start is too much
work. If you must set the breakpoints manually the source-before-file is
worthless in my eyes. And even if I don't restart the LLDB itself and
only the program I would like to debug the breakpoints are again
unresolved after re-start.

"unresolved" means that though lldb knows what address the breakpoint should be at, it wasn't able to put the breakpoint there yet. After a restart, all breakpoints are expected to be unresolved - since the libraries in question haven't been mapped into the address space of the program, and will remain so till the relevant library gets mapped in.

What do you mean with

since it knows the world is going to change out from under it

? The addresses of a loaded library doesn't change from run to run.

That's actually not true. If ASLR is on, then libraries load in different orders from run to run. And even if you turn ASLR off - which the debugger does by default - though all the directly loaded shared libraries will end up in the same place from run to run, the addresses of any libraries that are loaded later on as a result of user actions (e.g. hit Print in some OS X application and a bunch of shared libraries will get loaded) depend on the order of those actions.

Anyway, that's interesting maybe, but not really the point. At the beginning of _dyld_start, none of the shared libraries are mapped into the address space of the program. Since we need to write traps into the breakpoint memory locations in order to set breakpoints, we're going to fail until dyld gets a chance to map them in. The only thing that's been mapped in is the loader itself.

Jim

#7

Ok, I will file a bug.

Thank you for your help and your explanations.

Christian