Thread created / exited notifications

I was hoping that I could get events for threads being created and exiting.

Right now, it looks like this is sitting on top of a mountain of various issues. So, I’m hoping that I can go through some of the steps involved to see if I’m on track.

First up, is this something that would be useful to have?

It looks like that you basically have to get the current thread list when the process stops, and see how it differs from the last time you saw the thread list. This means that you miss out on any threads that were created and destroyed in between process stops.

One issue is platform availability for this information:

  • Windows: information about thread creation and destruction is provided directly by the Windows debug events. Right now, it is queued up and not processed until process stops though to maintain the current thread list.

  • Linux: it looks like you have to use thread_db, if it is available. (For example, if someone’s using musl libc, there is no equivalent functionality.) Thread lists are gotten via enumerating /proc/%d/task.

  • Mac OS X and iOS: it looks like this information is not available. Thread lists are gotten by calling Mach’s task_threads.

  • FreeBSD: there’s a thread_db, but this isn’t currently used. Right now, thread lists are gotten via PT_GETLWPLIST via ptrace.

LLDB/MI provides this information using async records:

  • =thread-created,id=“id”,group-id=“gid”

  • =thread-exited,id=“id”,group-id="gid"A thread either was created, or has exited. The id field contains the gdb identifier of the thread. The gid field identifies the thread group this thread belongs to.

So, looking at what we might need to do in LLDB:

  • We’d have a ThreadEventData.

  • We’d have an SBThread::GetThreadFromEvent().

  • For Targets where the stub supports these events, they would issue them as an asynchronous notification.

  • This asynchronous notification would get broadcast as an event.- For targets where the stub doesn’t support these, then perhaps some (new?) code in Process::UpdateThreadListIfNeeded() could broadcast created / exited events?

  • LLDB/MI could be updated to use the new events.

  • The GDB RSP support needs to be updated to handle the new event. It isn’t clear whether or not this is supported by GDB or if this would be an LLDB extension. (The GDB docs don’t mention this.)

  • Perhaps methods on SBProcess to allow OperatingSystem plugins to issue created / exited events.

Some questions:

  • First up, again, should we support thread creation / exit events?

  • When the target doesn’t provide this information, is it okay for them to be batched up as described above by diffing the thread list when updating the thread list?

  • If non-stop mode starts to work, how would the current model work where we just update a thread list at specific points in time? Wouldn’t thread creation / exit events be more appropriate at that point?

  • Any thoughts on how this might work or integrate with the Operating System plugin interface? (Right now, this is all about UpdateThreadList and doesn’t seem to provide a means for doing asynchronous notification like calling a method on SBProcess to create / exit threads…)

  • If extensions are made, is there anything for communicating this clearly to other people implementing things, like Facebook’s ds2?

It took about 10 hours to track through everything and read the relevant docs and look at other implementations (like Facebook’s ds2) to get a better idea for what’s going on in various areas here, but I may well have gotten something wrong … so any corrections are welcome. :slight_smile:

  • Bruce

Hello Bruce,

I've thought about this feature as well, but did not have the time to
implement it. My comments are inline.

I was hoping that I could get events for threads being created and exiting.

Right now, it looks like this is sitting on top of a mountain of various
issues. So, I'm hoping that I can go through some of the steps involved to
see if I'm on track.

First up, is this something that would be useful to have?

I think this would be useful piece of information that we should
display to the user. However, it is currently very low on my
priorities list.
On the other hand, if someone goes through the trouble of setting up
the general plumbing, i'd be happy to chip in with the linux support.

It looks like that you basically have to get the current thread list when
the process stops, and see how it differs from the last time you saw the
thread list. This means that you miss out on any threads that were created
and destroyed in between process stops.

Not necessarily. At least on linux we get notification on every thread
creation (and exit) and we can pass this information down quite
reliably.

One issue is platform availability for this information:

Windows: information about thread creation and destruction is provided
directly by the Windows debug events. Right now, it is queued up and not
processed until process stops though to maintain the current thread list.
Linux: it looks like you have to use thread_db, if it is available. (For
example, if someone's using musl libc, there is no equivalent
functionality.) Thread lists are gotten via enumerating /proc/%d/task.

On linux, we get this notification every time a thread is created (see
NativeProcessLinux::MonitorSIGTRAP). However, we currently just drop
it on the floor. The tricky part here is passing this information from
the remote stub, through the gdb-remote-protocol, and all the way to
the user.

Mac OS X and iOS: it looks like this information is not available. Thread
lists are gotten by calling Mach's task_threads.
FreeBSD: there's a thread_db, but this isn't currently used. Right now,
thread lists are gotten via PT_GETLWPLIST via ptrace.

LLDB/MI provides this information using async records:

=thread-created,id="id",group-id="gid"
=thread-exited,id="id",group-id="gid"A thread either was created, or has
exited. The id field contains the gdb identifier of the thread. The gid
field identifies the thread group this thread belongs to.

So, looking at what we might need to do in LLDB:

We'd have a ThreadEventData.
We'd have an SBThread::GetThreadFromEvent().
For Targets where the stub supports these events, they would issue them as
an asynchronous notification.

This asynchronous notification would get broadcast as an event.

For targets where the stub doesn't support these, then perhaps some (new?)
code in Process::UpdateThreadListIfNeeded() could broadcast created / exited
events?
LLDB/MI could be updated to use the new events.
The GDB RSP support needs to be updated to handle the new event. It isn't
clear whether or not this is supported by GDB or if this would be an LLDB
extension. (The GDB docs don't mention this.)
Perhaps methods on SBProcess to allow OperatingSystem plugins to issue
created / exited events.

Some questions:

First up, again, should we support thread creation / exit events?
When the target doesn't provide this information, is it okay for them to be
batched up as described above by diffing the thread list when updating the
thread list?

Sounds like a plausible fall-back solution, but I don't think that is
necessary. I would expect that the debug api on any system can report
thread creation events in a reliable fashion. That's just my
expectation though...

If non-stop mode starts to work, how would the current model work where we
just update a thread list at specific points in time? Wouldn't thread
creation / exit events be more appropriate at that point?

Yes, I think they would.

On some platforms (Linux) it looks like you have no choice but stop on new thread creation in order to debug the code run in the new threads. But on platforms where this is not necessary for debugging (Mac OS X) I would want this feature to be opt in. One of the goals for any debugger, it seems to me, is to make "continue" run the program as much like running without the debugger as possible. We need to stop at shared library loads in order to set new breakpoints, but other than that on OS X we don't really muck much with the process when it is running flat out.

BTW, though on OS X there isn't a system-provided way to learn of new thread creation, lldb has code to do this job by hand - by setting breakpoints on the two "thread start" routines that the system uses. I use these when running functions, so that we can enforce the "run only one thread" policy when either the function being run or the system in its wisdom decides to add threads to the program. We don't do thread destruction because there wasn't a real reason for doing it, but that could probably also be done with an appropriate breakpoint on the way out of these routines. But again, I'm not sure how useful this is in normal practice.

I would not be in favor of batch sending of new thread creation events on stop if you can't get live tracking of thread creation & destruction. It serves no useful purpose, since it is information you can easily gather yourself, and it gives the impression that you are getting accurate information beyond this simple calculation, which you aren't.

Jim

I was hoping that I could get events for threads being created and exiting.

Right now, it looks like this is sitting on top of a mountain of various issues. So, I'm hoping that I can go through some of the steps involved to see if I'm on track.

First up, is this something that would be useful to have?

It looks like that you basically have to get the current thread list when the process stops, and see how it differs from the last time you saw the thread list. This means that you miss out on any threads that were created and destroyed in between process stops.

One issue is platform availability for this information:

  • Windows: information about thread creation and destruction is provided directly by the Windows debug events. Right now, it is queued up and not processed until process stops though to maintain the current thread list.
  • Linux: it looks like you have to use thread_db, if it is available. (For example, if someone's using musl libc, there is no equivalent functionality.) Thread lists are gotten via enumerating /proc/%d/task.
  • Mac OS X and iOS: it looks like this information is not available. Thread lists are gotten by calling Mach's task_threads.
  • FreeBSD: there's a thread_db, but this isn't currently used. Right now, thread lists are gotten via PT_GETLWPLIST via ptrace.

LLDB/MI provides this information using async records:

  • =thread-created,id="id",group-id="gid"
  • =thread-exited,id="id",group-id="gid"A thread either was created, or has exited. The id field contains the gdb identifier of the thread. The gid field identifies the thread group this thread belongs to.

So, looking at what we might need to do in LLDB:

  • We'd have a ThreadEventData.
  • We'd have an SBThread::GetThreadFromEvent().
  • For Targets where the stub supports these events, they would issue them as an asynchronous notification.
    • This asynchronous notification would get broadcast as an event.
  • For targets where the stub doesn't support these, then perhaps some (new?) code in Process::UpdateThreadListIfNeeded() could broadcast created / exited events?
  • LLDB/MI could be updated to use the new events.
  • The GDB RSP support needs to be updated to handle the new event. It isn't clear whether or not this is supported by GDB or if this would be an LLDB extension. (The GDB docs don't mention this.)
  • Perhaps methods on SBProcess to allow OperatingSystem plugins to issue created / exited events.

Some questions:

  • First up, again, should we support thread creation / exit events?

I can be implemented as optional support, but we shouldn't make _anything_ that relies on it. We should probably have a call on SBProcess that tries to enable this feature. Something like:

SBError
SBProcess::EnableSynchronousThreadNotifications();

Macs would return an error, and any target that doesn't support it would return an error.

  • When the target doesn't provide this information, is it okay for them to be batched up as described above by diffing the thread list when updating the thread list?

Yes this could easily be done.

  • If non-stop mode starts to work, how would the current model work where we just update a thread list at specific points in time? Wouldn't thread creation / exit events be more appropriate at that point?

There is a whole bunch of work needed to get non-stop mode working. You would need your changes to detect new threads and report them. You would to start listening to SBThread events instead of listening to the process state changed events. Much much more. Each command that does anything to the process (continue, halt, kill) need to be updated to support this new mode. All thread based commands (stepping in/out/over/return, etc) need to be updated to try and deal with other threads that can be running. All ThreadPlans need to be completely reworked to deal with non-stop. Hitting breakpoints is a real pain in non-stop mode because when you hit a breakpoint you need to disable the BP, single step the thread, re-enable the BP and continue, but of course you will need to halt ALL threads when doing this (more thread plan modifications) so you don't miss the BP on other threads. The only correct way to enable non-stop mode in my opinion is to improve the EmulateInstruction classes for the current architecture to be able to emulate ALL opcodes for the current architecture so when you hit a breakpoint, you can emulate the original opcode using the read/write memory/read callbacks. In this case you wouldn't have to disable the BP, single step, and re-enable the BP, you would just emulate it and do the actual read/write reg/memory. This is the way dtrace does things. I know some people think non-stop mode is going to be enabled soon, but in reality it is a long way off and there is a ton of work to make it work correctly.

  • Any thoughts on how this might work or integrate with the Operating System plugin interface? (Right now, this is all about UpdateThreadList and doesn't seem to provide a means for doing asynchronous notification like calling a method on SBProcess to create / exit threads...)

We would send out thread events. lldb_private::Thread would make a new eBroadcastBitStateChanged bit just like lldb_private::Process does, and send out events for itself. Clients could listen for thread events and be able to catch them. The question is: do OS threads get treated just like actual threads? Do they send out events?

  • If extensions are made, is there anything for communicating this clearly to other people implementing things, like Facebook's ds2?

It took about 10 hours to track through everything and read the relevant docs and look at other implementations (like Facebook's ds2) to get a better idea for what's going on in various areas here, but I may well have gotten something wrong ... so any corrections are welcome. :slight_smile:

So overall the thread notifications might be nice to add using the correct format where you can listen to for thread created/exit notifications, maybe using a eBroadcastBitStateChanged bit just like lldb_private::Process and send a eStateLaunching state for threads being created and eStateExited for the thread exiting. Doing this only when the process stops seems like a good default (send out the new thread created/exited notifications). Doing anything synchronously seems like it will slow down debugging for no real gain as non-stop mode is really not close to being a reality...

Greg