SB API is not working properly with OSPython plugin

Hi lldb-dev,

I work on a custom implementation of OperatingSystem plugin using Python and SB API. I’m trying to fetch information about some variables from the target into the plugin, to do that I’m using the following Python code:

ready_tasks = self._target.FindGlobalVariables(‘pxReadyTasksLists’, 1).GetValueAtIndex(0)

When I do print(ready_tasks) I get:

No value

At the same time, doing the same actions inside lldb embedded interpreter follows to:

print(lldb.target.FindGlobalVariables('pxReadyTasksLists',1).GetValueAtIndex(0))

(List_t [5]) pxReadyTasksLists = {

[0] = {

uxNumberOfItems = 0

pxIndex = 0x00000000

xListEnd = {

xItemValue = 0

pxNext = 0x00000000

pxPrevious = 0x00000000

}

}

Does anybody know what may cause such a behavior?

All SBValues have an error in them (SBValue.GetError). Does that say anything interesting?

Jim

It is, the error is: error: error: process must be stopped.

I thought that the plugin (get_thread_info in my case) is invoked when the process is already stopped, but it’s not. Is it ok?

That’s a little complicated…

lldb has two levels of stop state - private stops and public stops. When the process gets notification from the underlying process plugin that the process stopped, it raises a private stop event. That gets handled by the ShouldStop mechanism on the private state thread in lldb, and then if the stop is judged interesting to the user, it gets rebroadcast as a public stop.

For instance, when you issue a “step” command, lldb will stop and start the process multiple times as it walks through the source line. But only the last of those stops are relevant to the user of LLDB, so all the other ones exist only as private stops.

The SB API’s for the most part should only consider a “publicly stopped” process accessible. After all, you wouldn’t want some API to succeed sometimes if you happen to catch it in the middle of a private stop.

But the OperatingSystem plugin needs to get called right after a private stop, so it can provide threads for the ShouldStop mechanism. We should really have some formal mechanism whereby things like the OS plugin can request elevated rights in the SB API’s, so that they can run at private stop time. IIRC, we instead have a hack where SB API calls that run on the private state thread are blanket allowed to run at private stop. The xnu Operating System plugin successfully gets global values to look up its threads. So I’m not sure why that isn’t working for you.

Can you cook up a simple example showing the failure and I’ll have a look?

Jim

Sure, could you describe in more detail which example may help you?

чт, 14 февр. 2019 г. в 22:36, Jim Ingham <jingham@apple.com>:

The simplest thing possible to reproduce the failure. So some OS_Plugin implementation which tries to look up a global like this and fails, and some program source I can run it under that provides said global. That way I can easily watch it fails as you describe when the plugin gets activated, and see why it isn’t allowing this call on private stop.

Jim

I found out that the plugin works well with an x86 application, so I think that the problem is in my process plugin. Maybe you know a place where to start looking for an issue?

Your plugin should have set the private state to stopped when it figures out however it does that the process has stopped. The API is Process::SetPrivateState. Is that happening at the right time?

Jim

It seems that the process plugin uses the Process::SetPrivateState at the right time. If you look at the log, you will see that the process is already in the 'private stopped’ state when the OS plugin is invoked.

(lldb) c
lldb Process::Resume – locking run lock
lldb Process::PrivateResume() m_stop_id = 1, public state: stopped private state: stopped
lldb Process::SetPrivateState (running)
intern-state Process::ShouldBroadcastEvent (0x1abea90) => new state: running, last broadcast state: running - YES
intern-state Process::HandlePrivateEvent (pid = 1) broadcasting new state running (old state stopped) to public
intern-state Process::PushProcessIOHandler pushing IO handler
intern-state Process::HandlePrivateEvent updated m_iohandler_sync to 1
lldb Process thinks the process has resumed.
intern-state timeout = , event_sp)…
lldb waited from m_iohandler_sync to change from 0. New value is 1.
dbg.evt-handler Process::SetPublicState (state = running, restarted = 0)
Process 1 resuming
lldb Process::SetPrivateState (stopped)
lldb Process::SetPrivateState (stopped) stop_id = 2
error: error: process must be stopped.
intern-state Process::ShouldBroadcastEvent (0x7f3e9c007450) => new state: stopped, last broadcast state: stopped - YES
intern-state Process::HandlePrivateEvent (pid = 1) broadcasting new state stopped (old state running) to public
intern-state timeout = , event_sp)…
dbg.evt-handler Process::SetPublicState (state = stopped, restarted = 0)
dbg.evt-handler Process::SetPublicState (stopped) – unlocking run lock

lldb Process::SetPrivateState (stopped) stop_id = 2

error: error: process must be stopped.

These two lines are printed from different threads, you cannot be sure the real order of execution is the same.

The plugin should subscribe on public state change events and wait until one comes (correct me if I’m wrong about that).

> lldb Process::SetPrivateState (stopped) stop_id = 2
> error: error: process must be stopped.

These two lines are printed from different threads, you cannot be sure the real order of execution is the same.

The plugin should subscribe on public state change events and wait until one comes (correct me if I’m wrong about that).

That's not right. When the process stops, but before lldb broadcasts a public stop event, it will query the OS Plugin directly to build up the thread list. It needs to do that before it declares a public stop because it uses the thread list to reason about whether to declare a public stop or not. So the OS Plugin (and BTW the Thread Step Plan plugin is in the same boat) have to run after a private stop but before public one. There isn't a principled way to do that. The best we have at present is a hack that says "if you are running on the private state thread, then you get to do things between private & public stop as if there were a public stop". Cleaning that up is item 5 on the lldb Projects page if anyone is interested...

Jim

That clarified things, thanks!

I think, this is the reason:

ProcessRunLock &Process::GetRunLock() {
  if (m_private_state_thread.EqualsThread(Host::GetCurrentThread()))
    return m_private_run_lock;
  else
    return m_public_run_lock;
}

In our case the current thread is not m_private_state_thread. I create a separate thread for waiting a processor to halt and then SetPrivateState. It seems, that was an inappropriate approach.

What's triggering one of the OS Plugin methods to get run on this separate thread? I would expect SetPrivateState would just cause the private stop event to get broadcast to the private state thread, and then that would wake up and then it would be the one to call the OS Plugin to do it's job. That's how the GDBRemote plugin works, for instance.

Jim

This is ThreadList::GetSelectedThread, it calls OperatingSystemPython::UpdateThreadList indirectly.

However, I cannot reproduce the error Alexander gets, due to a deadlock.