LLDB sends continue right after connection (gdb-remote)

Benjamin_Kemper · January 22, 2013, 2:55pm

Hi,

I’ve implemented a debugger backend that implements the gdb remote protocol, and I’ve noticed that while communication seems OK, right after the connection from LLDB to my implementation the process resumes automatically.

I’m guessing that this is not the expected behaviour, but when looking at the logs I did see LLDB sending the “$vCont” packet without any user interaction.

I used the following command to connect:
process connect -p gdb-remote connect://localhost:52168

And this is an excerpt from the logs:

ProcessGDBRemote::UpdateThreadList (pid = 1799)
0x7fe2aa2268b0: ThreadGDBRemote::ThreadGDBRemote (pid = 1799, tid = 0x0707)
< 22> send packet: $qThreadStopInfo707#99
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 1
< 1> read packet: +
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 4
< 4> read packet: $#00
< 1> send packet: +
error: failed to get response for ‘qThreadStopInfo707’
ProcessGDBRemote::Resume()
ProcessGDBRemote::AsyncThread (arg = 0x7fe2ab00d000, pid = 1799) Got an event of type: 1…
ProcessGDBRemote::AsyncThread (arg = 0x7fe2ab00d000, pid = 1799) got eBroadcastBitAsyncContinue: vCont;c:0707
GDBRemoteCommunicationClient::SendContinuePacketAndWaitForResponse ()
GDBRemoteCommunicationClient::SendContinuePacketAndWaitForResponse () sending continue packet: vCont;c:0707
< 16> send packet: $vCont;c:0707#b0
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 1
< 1> read packet: +
GDBRemoteCommunicationClient::SendContinuePacketAndWaitForResponse () WaitForPacket(vCont;c:0707)
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xffffffff, status = success, error = (null)) => bytes_read = 7
< 7> read packet: $W00#b7
< 1> send packet: +
GDBRemoteCommunicationClient::SendContinuePacketAndWaitForResponse () got packet: W00
GDBRemoteCommunicationClient::SendContinuePacketAndWaitForResponse () => exited
ProcessGDBRemote::AsyncThread (arg = 0x7fe2ab00d000, pid = 1799) thread exiting…

Any ideas why this is happening?

Thanks,
Benjamin.

Malea_Daniel · January 22, 2013, 3:55pm

I’m not too familiar with the GDB remote plugin, but it looks like ProcessGDBRemote::Resume() is being called, which is somewhat surprising because at first glance it doesn’t look like anything in UpdateThreadList() should be calling that.

If you can, I’d recommend making a debug build of lldb and starting it with another debugger (gdb) attached and set a breakpoint in ProcessGDBRemote::DoResume() to see who calls it and why.

If you find out more information, feel free to log a defect at http://llvm.org/bugs/enter_bug.cgi?product=lldb

Cheers,

Dan

Greg_Clayton1 · January 22, 2013, 5:53pm

I believe we have a bug where if you attach and the stop reply packet that we send immediately after the attach (the '?' packet) comes back with a T00 (meaning the thread stopped with signal 0, or no signal), we end up resuming the process. We normally get a T11 (SIGSTOP) when we first attach on MacOSX. This gives LLDB a reason for why the process stopped. It is also helpful if you add the stop reply packet extensions we have documented in the "lldb-gdb-remote.txt" file in our SVN repository:

svn cat http://llvm.org/svn/llvm-project/lldb/trunk/docs/lldb-gdb-remote.txt

See the section labeled "Stop reply packet extensions". The standard stop reply packet is really UNIX centric and it assumes every stop reason can be explained by a signal. We extended the contents of the key/value pairs that are returned to provide more complete information.

Can you enable GDB packet logging and send me the output? I will be able to tell more from the log.

To enable this logging:

(lldb) log enable -f /tmp/packets.txt gdb-remote packets
(lldb) … < connect to remote process > …
(lldb) quit

Then attach and send the packet.txt file.

Greg

Benjamin_Kemper · January 23, 2013, 5:24pm

Hi Greg,

Thanks, this seems like a very plausible reason.

Two questions:

Do I need to implement all the stop reply packet extensions (In the description it says that they are good for performance but no necessary) or just qThreadStopInfo?
I’ve looked at the ? command in my logs and saw that I’m receiving something different than T00, is this value Ok or also bad?

The log:

< 5> send packet: $?#3f
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 1
< 1> read packet: +
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 52
< 52> read packet: $T0206:0*,;07:f08a5851ff7f0* ;10:2880276eff7f0* ;#67
< 1> send packet: +

Thanks for the help,
Benjamin.

Greg_Clayton1 · January 23, 2013, 6:11pm

Hi Greg,

Thanks, this seems like a very plausible reason.

Two questions:
1. Do I need to implement all the stop reply packet extensions (In the description it says that they are good for performance but no necessary) or just qThreadStopInfo?

The stop reply packet and qThreadStopInfo return the exact same data. The stop reply packet is basically the qThreadStopInfo for the most important thread that stopped for a reason (or any thread that stops for a reason, LLDB doesn't care).

The shortcoming in the GDB remote protocol is the idea that one and only one thread stops for a reason at any given time. This simply isn't try on darwin, and probably not true for many other systems.

2. I've looked at the ? command in my logs and saw that I'm receiving something different than T00, is this value Ok or also bad?

The log:
< 5> send packet: $?#3f
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 1
< 1> read packet: +
size_t GDBRemoteCommunication::WaitForPacketWithTimeoutMicroSecondsNoLock(StringExtractorGDBRemote &, uint32_t): Read (buffer, (sizeof(buffer), timeout_usec = 0xf4240, status = success, error = (null)) => bytes_read = 52
< 52> read packet: $T0206:0*,;07:f08a5851ff7f0* ;10:2880276eff7f0* ;#67
< 1> send packet: +

LLDB doesn't currently handle the '*' in the expedited register values.

You also have issues with your stop reply packet:
1 - "06:0*,;" has a comma in the value? This seems wrong.
2 - there is an extra space which shouldn't be there ("07:f08a5851ff7f0* ;" and "10:2880276eff7f0* ;"
3 - there is no "thread" key value pair to say which thread this data is for. This hex thread ID value must be in the packet or the packet doesn't make much sense.

Here is a sample from the LLDB debugserver:

$T11thread:1c03;qaddr:a0;threads:1c03;02:0000000000000000;03:0000000000000000;04:0000000000000000;05:0000000000000000;06:0000000000000000;07:e8f7bf5fff7f0000;08:0000000000000000;09:0000000000000000;10:2810c05fff7f0000;11:0002000000000000;metype:5;mecount:2;medata:10003;medata:11;#00

A few things to note:

"thread:1c03;" is specified first
"qaddr" is not needed on non-darwin platforms
"threads:1c03;" is a key/value pair that can supply all the thread IDs for all threads so we don't have to send the "qfThreadInfo" and "qsThreadInfo" packets/
All registers are expanded to full width. Feel free to file a bugzilla bug to have LLDB Support the '*' feature.
"metype", "mecount", and "medata" key value pairs are only needed for mach kernels.

So to get LLDB happy with your packets:
- add a "thread" to every stop reply packet/qThreadStopInfo
- use the same output for stop reply packets and qThreadStopInfo
- expand your register values (don't use the '*')
- Fix your register values to omit extra spaces and commas
- if you return T00, be sure to add a "reason" key/value pair if the thread is stopping for a reason that isn't actually signal related.

Thanks for the help,

Let me know if you have any questions.

Greg

Benjamin_Kemper · January 24, 2013, 11:46am

Thanks Greg!

Adding the “thread” pair and uncompressing the register values did the trick.