Understanding debugger launch events sequence

Hi,

On mac OS, I am having difficulty understanding the launch debugger events sequence of lldb. I used the following code to play around LLDB. I found, for some binaries, debugger will enter stopped/paused mode, waiting for my further input, print stack shows:

bt

  • thread #1: tid = 0x15153e, 0x00007fff5fc0d2af dyld`gdb_image_notifier(dyld_image_mode, unsigned int, dyld_image_info const*) + 1
  • frame #0: 0x00007fff5fc0d2af dyld`gdb_image_notifier(dyld_image_mode, unsigned int, dyld_image_info const*) + 1
    frame #1: 0x000000000000401d

But some other binaries, it just print “Process event: stopped, reason: 1” and inferior just exits immediately without waiting for debugger’s further input.

Questions:

  1. When I launch a binary, is there supposed to be a loader breakpoint waiting for debugger continue? Any other debug events do I expect to get and continue?
  2. What about attach?
  3. What is the dyld`gdb_image_notifier() debugger break above? Why does it happen for some binary but not others?

Thanks for any information!

Should be first for LLDB package to be added to search path.

from find_lldb import lldb

from lldb import eStateStepping, eStateRunning, eStateExited, SBBreakpoint, SBEvent, SBListener, SBProcess, SBTarget
import sys
import os
import subprocess
from sys import stdin, stdout
from threading import Thread

class LLDBListenerThread(Thread):
should_quit = False

def init(self, process):
Thread.init(self)
self.listener = SBListener(‘Chrome Dev Tools Listener’)
self._add_listener_to_process(process)
self._broadcast_process_state(process)
self._add_listener_to_target(process.target)

def _add_listener_to_target(self, target):

Listen for breakpoint/watchpoint events (Added/Removed/Disabled/etc).

broadcaster = target.GetBroadcaster()
mask = SBTarget.eBroadcastBitBreakpointChanged | SBTarget.eBroadcastBitWatchpointChanged | SBTarget.eBroadcastBitModulesLoaded
broadcaster.AddListener(self.listener, mask)

def _add_listener_to_process(self, process):

Listen for process events (Start/Stop/Interrupt/etc).

broadcaster = process.GetBroadcaster()
mask = SBProcess.eBroadcastBitStateChanged
broadcaster.AddListener(self.listener, mask)

def _broadcast_process_state(self, process):
state = ‘stopped’
if process.state == eStateStepping or process.state == eStateRunning:
state = ‘running’
elif process.state == eStateExited:
state = ‘exited’
self.should_quit = True
thread = process.selected_thread
print ‘Process event: %s, reason: %d’ % (state, thread.GetStopReason())

def _breakpoint_event(self, event):
breakpoint = SBBreakpoint.GetBreakpointFromEvent(event)
print ‘Breakpoint event: %s’ % str(breakpoint)

def run(self):
while not self.should_quit:
event = SBEvent()
if self.listener.WaitForEvent(1, event):
if event.GetType() == SBTarget.eBroadcastBitModulesLoaded:
print ‘Module load: %s’ % str(event)
elif SBProcess.EventIsProcessEvent(event):
self._broadcast_process_state(SBProcess.GetProcessFromEvent(event))
elif SBBreakpoint.EventIsBreakpointEvent(event):
self._breakpoint_event(event)

def _interctive_loop(debugger):
process = debugger.GetSelectedTarget().process
event_thread = LLDBListenerThread(process)
event_thread.start()

while (True):
stdout.write('dbg> ')
command = stdin.readline().rstrip()
if len(command) == 0:
continue
debugger.HandleCommand(command)

def main():
debugger = lldb.SBDebugger.Create()

print(‘Working Directory: %s’ % os.getcwd())
debugger.HandleCommand(‘target create /usr/bin/find’)
debugger.HandleCommand(‘run .’)
_interctive_loop(debugger)

if name == ‘main’:
main()

Hi Jeffrey,

I see a couple of problems with the way you are using the lldb's API.
The main problem is you are launching the target via the command-line
API, which does not allow you to specify the listener upon creation.
When you start it this way all events go to the default debugger
listener (debugger.GetListener()), and by the time you connect your
own listener, some of these events have already been broadcast, and
that is why you get nondeterministic behavior. You should use the
SBTarget.Launch function to specify the listener from the start.

The second problem is the handling of the Stopped events. Sometimes
LLDB needs to stop the inferior do to some internal work, but this the
program is immediately resumed. This event is broadcast as a "stopped"
event with a special "restarted" bit set (see
SBProcess.GetRestartedFromEvent, and
<http://lists.llvm.org/pipermail/lldb-dev/2016-January/009291.html&gt;\)

hope that helps,
pl

Great, this is very helpful, will give a try.
Btw: is there any threading requirement of lldb API? For example, are all the Apis must be called on the event thread or they are free to be called on any thread? I know windows debug API has some limitation on this.

There is no requirement that the lldb API’s be called on a particular thread on OS X. LLDB tries to be robust against being called from multiple threads simultaneously for the same debugger, but you can still make it fall over if you try hard, particularly if you allow multiple threads to restart the process you are debugging. Running multiple SBDebuggers on separate threads works fine, that’s the mode Xcode uses, and we haven’t had problems with this in quite a while.

Jim

Thanks Jim. Is this true for other platforms? Our IDE is going to support Mac and Linux and may extend to Windows some time later.
Just curious, why does Xcode create multiple SBDebuggers assuming it is debugging a single process? Are you talking about multiple-processes scenario(One SBDebugger for one process)?

I can’t comment on Windows, I don’t know what requirements the Windows API’s place on the debugger.

Its been a while since I’ve worked on Linux, but I don’t remember anything that would privilege one thread over another.

lldb supports running multiple targets and processes in one debugger, and also supports multiple debuggers running each one target or any combination. Since each Debugger gets a separate script interpreter (and all its state) by running multiple processes in one SBDebugger you could offer users the possibility of having scripted commands to control a set of processes (e.g. hitting a breakpoint in one process could trigger actions in the other.) It might be possible to do some interesting things that way.

OTOH, keeping to one process per debugger is a much simpler programming model. So if you were planning to have YOUR code that runs the debugger handle the possible interactions among processes, then it is probably going to be easier to manage doing it that way.

Jim

Jim/Pavel, my toy code works reliably after using SBListener with SBTarget.Launch/Attach.
One thing I noticed is:
If I set “stop_at_entry=True” for SBTarget.Launch(), I will get a stop event of eStopReasonSignal at loader breakpoint. However SBTarget.AttachXXX() will pause the target process without a stop event. Is this expected? This may cause a bit state issue in our IDE since we rely on the stop event from debugger to update UI in IDE. Is there any way to tell lldb to emit a stop event during attach?

I don’t think we can change this behavior, since other clients are relying on the way it is now.

In any case, attach won’t return till it is successful, and presumably you know you are attaching, so I don’t think there’s any ambiguity about what is going on, even if you don’t get a stop event.

Jim

Thanks. That’s fine, just want to confirm the behavior and my understanding. I can definitely deal with the difference between attach/launch myself.

AFAIK, windows spawns a separate thread to do the actual debugging,
but this is hidden from you, and you can carry on debugging normally.
On Linux/OSX we even spawn a separate process.

In short, there should be no noticeable user-facing differences
between the platforms. If you find any, then we'd like to know about
them.

pl

Great, thanks for the confirmation.