gdb-remote to android gdbserver

Hi,

I have started to investigate connecting lldb to gdbserver running on an android device (arm). It connects correctly, I’m able to start the remote program and interrupt it but lldb doesn’t permit me to access any data as it says the current frame is invalid:

mbp:NativeTest meeloo$ lldb …/…/…/ndk/samples/hello-gl2/libs/armeabi/libgl2jni.so
Current executable set to ‘…/…/…/ndk/samples/hello-gl2/libs/armeabi/libgl2jni.so’ (arm).
(lldb) gdb-remote
error: gdb-remote [:]
(lldb) gdb-remote 127.0.0.1:5039
Process 12260 stopped

  • thread #1: tid = 0x2fe4, , stop reason = signal SIGTRAP
    frame #0:
    (lldb) frame variable
    error: invalid frame
    (lldb) continue
    Process 12260 resuming
    ---- Here I hit Ctrl+C
    Process 12260 stopped
  • thread #1: tid = 0x2fe4, , stop reason = signal SIGINT
    frame #0:
    (lldb) continue
    Process 12260 resuming
    (lldb) process interrupt
    Process 12260 stopped
  • thread #1: tid = 0x2fe4, , stop reason = signal SIGINT
    frame #0:
    (lldb) register read
    error: invalid frame
    (lldb) process status
    Process 12260 stopped
  • thread #1: tid = 0x2fe4, , stop reason = signal SIGINT
    frame #0:

Does anyone have already tried to do this kind of things? I’m trying to get it to work before moving to implementing android support in my program with the C++ API…

Thanks in advance,

S.

Actually when I checked if it worked or not out of the box, I noticed one missing thing was RLE (run-time length encoding) support in GDB remote protocol.
I have a patch for that (change is quite small), that I can clean up and upload soon if you are interested.

Yes, I’m very much interested!

Thanks Virgile,

S.

Yes, please do submit the RLE patch.

The problem is we don't know the registers. GDB assumes that the GDB that is being used knows what registers are on the other side of the remote connection. New GDB binaries and GDBSERVER binaries are compiled for each target. LLDB has one binary for all systems. So LLDB expects to be able to query the GDBSERVER for the register information. LLDB has defined new packets to get this information. We have documented the packets we added:

svn cat http://llvm.org/svn/llvm-project/lldb/trunk/docs/lldb-gdb-remote.txt

I believe there is a way defined in the remote protocol to query for registers using the "qXfer" command with the "xmlRegisters" gdb feature. We haven't added support for this, but the "XML target description" format is missing information we need. The details on this are at:

https://sourceware.org/gdb/onlinedocs/gdb/Target-Description-Format.html#Target-Description-Format

It could also be easily extended to support giving us the information required.

So, in order to get things going you will need to do one of the following:
1 - add support for the qRegisterInfo packet (easiest)
2 - add a new "settings set" setting for the "gdb-remote" plug-in that contains a path to a register description file. The file format should be something parseable like XML and you would need to write a parser for it and parse the contents by adding a method to the GDBRemoteDynamicRegisterInfo class (harder)
3 - If the current gdbserver for android does support the "XML target description", we would need to modify the GDBRemoteCommunicationClient class to be able to retrieve the information and parse the XML. You would then need to modify the gdbserver for android to add extra data we need to the register info (compiler register numbers for each register if one exists, DWARF register numbers, generic register info) (hardest)

I think option 1 or 2 would be the easiest.

Greg

If I understand correctly 1 and 3 implies modifying the android gdbserver while 2 could be done entirely in lldb? Having gdbserver changes accepted by the maintainers and integrated in future builds of android and/or the NDK looks like a hard path to walk.
Also I have looked at the gdbserver sources of the current android (found here https://android.googlesource.com/toolchain/gdb/ ) and a couple of greps seems to indicate that xmlRegisters is only supported on x86 although register definition xmls exists for many arm/linux variants.
In this light 2 seems like a good short term solution to force a register mapping on lldb by reading the xml from the gdb sources, until 1 is finally implemented in both projects and lldb is able to detect the register mapping.

Is that correct?

S.

Yes, I suppose a lldb-only change would be much more easier for now.

From what I remember when I tested, hard-coding the ARM registers temporarily + the runtime-length encoding patch (that I will commit soon) was enough to to have a simple address breakpoint working.
Backtrace didn’t, but I didn’t bother investigate further.

If I understand correctly 1 and 3 implies modifying the android gdbserver while 2 could be done entirely in lldb?

yes

Having gdbserver changes accepted by the maintainers and integrated in future builds of android and/or the NDK looks like a hard path to walk.
Also I have looked at the gdbserver sources of the current android (found here https://android.googlesource.com/toolchain/gdb/ ) and a couple of greps seems to indicate that xmlRegisters is only supported on x86 although register definition xmls exists for many arm/linux variants.
In this light 2 seems like a good short term solution to force a register mapping on lldb by reading the xml from the gdb sources

We will need an XML file with all the right data, which is more than what is in the GDB sources. We need compiler register numbers (for EH frame decoding) and DWARF register numbers for debug info parsing, and generic register info (which reg is the PC, SP, FP, etc). The compiler and DWARF register numbers don't always match (i386 has EBP and ESP different between the compiler and DWARF reg nuns).

, until 1 is finally implemented in both projects and lldb is able to detect the register mapping.

Sure.

Is that correct?

Yes.

So the theory would be:

1 - Create an XML register definition file format (or modify the GDB one to include the data we need)
2 - set the setting in LLDB prior to connecting:
  (lldb) settings set plugin.process.gdb-remote.register-definition-file /tmp/registers.xml
3 - run and debug and LLDB would use the registers defined in "/tmp/registers.xml" if qRegisterInfo packet is not supported by the GDB server we connect to.

This this means adding a setting and modifying GDBRemoteDynamicRegisterInfo to have a new method that can import register data from the XML file from step 1.

Greg

On Darwin, r7 is used as the frame pointer. I think other platforms typically use r11? That could explain the backtracing problems right there. The g_register_infos table in the ABIMacOSX_arm.cpp ABI plugin and ABIMacOSX_arm::CreateDefaultUnwindPlan().

I have been working in the last few weeks on making LLDB working with gdbserver on Linux host. Some of the things that I noted missing are.

  1. Run length encoding as already mentioned. I have a patch for it that I will submit shortly.

  2. I had to hardcode the registers for x86_64 as gdbserver does not support qRegisterInfo.

  3. Make sure to use g/G packets for registers.

After that I had a working connection. But then I noticed the following 2 problem.

PC is not decremented by length of breakpoint instruction after target stops.

LLDB failing to recognize a breakpoint is for stepping and stopping before a source level step was really complete.

I am working on these 2 issues.

Regards,

Abid

Hello,

Sorry, since I also had a RLE patch pending, here it is in case it is useful (before it’s too late).
Hope we can merge both of them together maybe?

This one also tries to handle escaped characters (#, $ and }), when receiving and sending.

Did it from http://sourceware.org/gdb/onlinedocs/gdb/Overview.html
Also, it converts everything where the packet copy was happening, to avoid extra copy and O(n) insertion.

Not sure if checksum is supposed to work on the escaped version or not. Need to check how GDB is working (spec doesn’t say anything).

I didn’t have a chance to test it again since I added character escaping, so I should probably do that again in case there is real interest in it.
Also couldn’t polish it as I intended, but thought it would still be more useful as is before the other one get merged than after.

Sincerely,
Virgile

lldb-rle.diff (3.3 KB)

Virgile,

Thanks for the patch. You choose a better place to decode the RLE. I am not sure about escape character though as gdbserver does not seem to use them for normal packets. The data byte is encoded in as 2-digit asci character. For example, this is a response to a g packet request.

putpkt ("$2a0*}0*)40e2f*"7f0S20hb0b6ddf7ff7f0*"020* 330*“2b0*}0*}0* 7f030*(f* 0*}0*}0*}0*}0*}0c801f0 3b0*}0*}0*}0*}0*}0*E#c8”); [noack mode]

and initial part of how gdb decoded it.

2a0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040e2

I will test the RLE part of your patch. If that is ok then I will merge it.

Thanks,

Abid

Great, thanks!

Hope it will work out, esp. the escaping at encoding time that has never been tested (but well, if there is no special characters it should be fine).
If special characters proves to be useless or causing issues, I have a much smaller version just doing the RLE part (basically, encoding part disappear and decoding just test for “*”).

Virgile

I committed part of your patch that handles the RLE after cleaning it a bit and testing. I noticed though that expanding RLE at this point has a little problem. Packet log shows the un-expanded packet. I will fix it separately.

Regards,

Abid

Great, thanks for testing/cleaning it!

Testing with qemu, I observed that we had not cleared the ‘packet_str’ before starting putting data in it. It can cause it to grow and fail checksum test. Sending out a fix for it.