LLDB for Android initiative

Subprojects LLDB
1

Hi all!

I’m starting up an effort to get LLDB running on Android. I just wanted to reach out, say hi, and give you an outline of how I’m thinking about attacking this effort. I’m looking for feedback, so please fire away if you have any suggestions or comments!

I’m thinking of attacking the effort in stages, looking something like this:

  1. Get LLDB up and running against a local Linux x86 process.

It looks like many aspects of this already work. I’ve heard there might be some rough edges around core dump support, DWARF 4/5 support, and possibly some optimized debug info support on the clang side, so any work here might touch those areas.

I see we have what looks like 2 buildbots dedicated to building lldb in linux scenarios:

http://lab.llvm.org:8011/builders/lldb-x86_64-debian-clang
http://lab.llvm.org:8011/builders/lldb-x86_64-linux

Android currently builds linux host tools as 32-bit. Both of those buildbots above appear to be 64-bit. I’d love to get the equivalent of an Ubuntu 12.04 LTS x64 buildbot building a 32-bit LLDB executable. How can I go about setting that up?

  1. Get the LLDB remote solution up and running against a remote Linux x86 process.

Here we get to the first high-level question mark: do we continue to use gdbserver, use debugserver, or base something on lldb-platform? I haven’t dug into this yet. I’ve heard some thoughts on this topic, such as (a) LLDB has extended the gdb remote protocol and offers some benefits over using gdbserver, (b) debugserver is currently very part-specific and might be a painful way to go in the short term (but I haven’t heard comments on the longer-term potential benefits of toughing through that), and (c) lldb-platform is a reasonable starting point and has been used to get some traction bringing up LLDB on other chipsets. Like in (1), I’ll want to set up a build bot that builds and runs remote tests in this environment.

Any thoughts on this?

  1. Get the LLDB remote solution up and running against a remote Linux ARM system.

The idea being that it will be easier for me to poke around on the Linux ARM system than it would be to go straight for the Android device or emulator, but gets me working against an ARM system, one step closer to a typical Nexus device. And helps out ARM Linux remote support in the process (if there are any weak spots). I don’t know yet what the scope of work here might entail. Similar to (2), I’ll want to set up a build bot that builds and runs remote tests in this environment as well.

  1. Getting LLDB remote solution up and running against an Android ARM device.

  2. Either directly implement or make it straightforward for Android vendors to fill in anything necessary to use our remote solution on other Android hardware.

I look forward to working with the LLDB community on this effort! Suggestions or comments are appreciated.

Sincerely,
Todd Fiala

2

Hi Todd,

It’s great that you are looking at lldb!

In general, it looks like many of the things you want to do with lldb right now you can, to some extent. I don’t think there are too many people looking at lldb for Linux right now as the Linux buildbots are failing so there may be bugs introduced as time goes on. Also, the only real way to do some remote debugging with Linux today is to use gdbserver. The major feature lldb for Linux (and useful for lldb in general) would benefit from is a “lldbserver” that works for all platforms.

I’ve also provided some comments inlined below…

3

Thanks for the reply, Matt!

4

Hi Todd,

I work on a sizable cross-platform mobile app that uses the Android NDK pretty heavily. I’ve had very hit or miss experiences debugging with gdb in that environment, so I think this is a great idea.

Have you started a branch for this effort or anything like that? I would be very interested in helping improve the NDK debugging experience in any way.

Sincerely,
-Alex Weisberger

5

Hi Alex,

Thanks for the offer!

If my work is successful, AOSP will end up with an LLDB built from that work. I’m planning on working upstream for the real work, and I intend to keep patches as bite-sized as possible to reduce latency on upstream approval. So I think the bulk of the work will be in the top of tree unless something makes that impractical. We’re trying to improve the native debugging experience on Android, and this is going to be one of our investments there.

Regarding the NDK and debugging, there was an OTA upgrade issue to Android 4.3 where the Android device’s run-as command (/system/bin/run-as) would lose its setgid/setuid permissions. This would render gdbserver unable to attach to the device process, making (as you would guess) remote debugging impossible. One of the first things I did when I joined Google was to add a CTS test to verify run-as works. This issue was one of the more common issues I saw. It gets fixed by doing an over-the-USB cable re-imaging if you’re on a Nexus device, or whatever your vendor requires for system imaging otherwise. This should be resolved in 4.4.

Thanks for your interest in this work!

Sincerely,
Todd Fiala

6

Hi Todd,

I did some experimental work back in September to see what the road blocks might be in getting LLDB working with Android. I’m attaching a patch from the branch I was working on back then. This patch is probably badly out-of-date now, and it only just wiggled even when it was fresh. It might be of some use to you though.

The patch contains several things of varying degrees of usefulness.

  1. It implements LLDB “platform” support based on ADB. The idea behind this was to see if I could get LLDB to attach to a “clean” Android device and install whatever bits were necessary to connect and start debugging. This sort of worked but it’s quite sloppy, has almost no error handling and it’s debatable how much value it provides. Also, this was my first exploration of LLDB’s platform mechanism, which was relatively new at the time, and I can’t promise I was doing it right.

  2. It provides a set of hard-coded x86-based register definitions. This is absolutely not the correct way to do this, but it was the quickest way to get something working. You’d think you could reuse one of the existing register context classes provided by LLDB, but the implementation that was current at the time didn’t work that way (and I think it still doesn’t). The register definition mechanism that Matt referred you to is much better.

  3. It fixes a problem with LLDB’s remote protocol handling that causes it to throw away register values provided in a gdb-remote packet if the values don’t appear in the expected order. The particular problem this fixed for me was that the version of gdbserver I was working with was sending stop packets with values for the stack and instruction pointer registers, but they appeared before the thread info and so LLDB was ignoring them. This was compounded by the fact that the version of gdbserver I was using wasn’t responding to attempts to query the value of these registers directly (or rather it was responding but it wasn’t giving me the values).

With this patch, I was able to attach to a process running on an x86 emulator. As I recall I wasn’t able to set a breakpoint. I didn’t get as far as figuring out why, but I suspect it was something simple.

I definitely think that the platform independent remote debugging proposal that Matt referred you to is the correct direction for future work, but getting things working with gdbserver is probably at least an interesting exercise.

Good luck!

-Andy

android-lldb.patch (48.9 KB)

7

Hey, awesome Andy!

Thanks for the patch and the details. I’m pretty sure I’ll be interested in trying to at least get the current Android gdbserver working with lldb if for no other reason than to help decouple the remote platform piece from any other work that could be done in parallel on the lldb side. I’m sure this will be useful info (and maybe code too) to have in that pursuit. Very much appreciated!

The register definition mechanism that Matt referred you to is much better.

I definitely have a task on my list to look at that. Glad to hear others have given the idea a once-over, too.

Sincerely,
Todd Fiala

8

Hi all!

I'm starting up an effort to get LLDB running on Android. I just wanted to reach out, say hi, and give you an outline of how I'm thinking about attacking this effort. I'm looking for feedback, so please fire away if you have any suggestions or comments!

I'm thinking of attacking the effort in stages, looking something like this:

1. Get LLDB up and running against a local Linux x86 process.

It looks like many aspects of this already work. I've heard there might be some rough edges around core dump support, DWARF 4/5 support, and possibly some optimized debug info support on the clang side, so any work here might touch those areas.

I see we have what looks like 2 buildbots dedicated to building lldb in linux scenarios:

http://lab.llvm.org:8011/builders/lldb-x86_64-debian-clang
http://lab.llvm.org:8011/builders/lldb-x86_64-linux

Android currently builds linux host tools as 32-bit. Both of those buildbots above appear to be 64-bit. I'd love to get the equivalent of an Ubuntu 12.04 LTS x64 buildbot building a 32-bit LLDB executable. How can I go about setting that up?

I believe Matt Kopec covered this already.

2. Get the LLDB remote solution up and running against a remote Linux x86 process.

Here we get to the first high-level question mark: do we continue to use gdbserver, use debugserver, or base something on lldb-platform? I haven't dug into this yet. I've heard some thoughts on this topic, such as (a) LLDB has extended the gdb remote protocol and offers some benefits over using gdbserver, (b) debugserver is currently very part-specific and might be a painful way to go in the short term (but I haven't heard comments on the longer-term potential benefits of toughing through that), and (c) lldb-platform is a reasonable starting point and has been used to get some traction bringing up LLDB on other chipsets. Like in (1), I'll want to set up a build bot that builds and runs remote tests in this environment.

Any thoughts on this?

For the remote solution I would propose you do the following:
- Modify the gdbserver binary to support the extended LLDB packets. All these packets are described here:

svn cat http://llvm.org/svn/llvm-project/lldb/trunk/docs/lldb-gdb-remote.txt

- Compile the "lldb-platform" binary for x86. This is a binary that can get/put files, install executables, make directories and symlinks, list remote processes, attach and launch remote processes and much much more. It currently uses the GDB remote protocol and all you really need to do is to get it to build for linux. After this you can start the lldb-platform binary and then attach to it from a remote LLDB. I am currently working on a large patch that will be committed in the next few days which exposes the platform through the API as lldb::SBPlatform objects.

If you get the gdbserver updated to handle the extra packets, this should get you debugging in no time on Android. The platform will get you the rest of the debugging experience (auto install, remote shell commands, and much much more.

3. Get the LLDB remote solution up and running against a remote Linux ARM system.

The idea being that it will be easier for me to poke around on the Linux ARM system than it would be to go straight for the Android device or emulator, but gets me working against an ARM system, one step closer to a typical Nexus device. And helps out ARM Linux remote support in the process (if there are any weak spots). I don't know yet what the scope of work here might entail. Similar to (2), I'll want to set up a build bot that builds and runs remote tests in this environment as well.

My new upcoming platform patches will allow you to remotely run the test suite on there remote devices using the lldb-platform. Everything for ARM is the same as the x86 solution above: modify the gdbserver to support the LLDB packets and get the lldb-platform to compile for the remote ARM system.

4. Getting LLDB remote solution up and running against an Android ARM device.

Same flow as linux ARM.

5. Either directly implement or make it straightforward for Android vendors to fill in anything necessary to use our remote solution on other Android hardware.

I think we can grow the SBPlatform and the internal lldb_private::Platform to do a lot of great stuff. As soon as my patch lands, please take a look and see what you think.

If you have any questions, ask away.

Greg Clayton

9

Thanks for all the pointers and suggestions, Greg!

More below.

10

Thanks for all the pointers and suggestions, Greg!

More below.

... stuff deleted...
>
> 2. Get the LLDB remote solution up and running against a remote Linux x86 process.
>
> Here we get to the first high-level question mark: do we continue to use gdbserver, use debugserver, or base something on lldb-platform? I haven't dug into this yet. I've heard some thoughts on this topic, such as (a) LLDB has extended the gdb remote protocol and offers some benefits over using gdbserver, (b) debugserver is currently very part-specific and might be a painful way to go in the short term (but I haven't heard comments on the longer-term potential benefits of toughing through that), and (c) lldb-platform is a reasonable starting point and has been used to get some traction bringing up LLDB on other chipsets. Like in (1), I'll want to set up a build bot that builds and runs remote tests in this environment.
>
> Any thoughts on this?

For the remote solution I would propose you do the following:
- Modify the gdbserver binary to support the extended LLDB packets. All these packets are described here:

svn cat http://llvm.org/svn/llvm-project/lldb/trunk/docs/lldb-gdb-remote.txt

Ok - I'll have a look at that. Thanks for the link! How I approach this area might depend on the mechanics of getting changes into gdbserver.

- Compile the "lldb-platform" binary for x86. This is a binary that can get/put files, install executables, make directories and symlinks, list remote processes, attach and launch remote processes and much much more. It currently uses the GDB remote protocol and all you really need to do is to get it to build for linux. After this you can start the lldb-platform binary and then attach to it from a remote LLDB. I am currently working on a large patch that will be committed in the next few days which exposes the platform through the API as lldb::SBPlatform objects.

If you get the gdbserver updated to handle the extra packets, this should get you debugging in no time on Android. The platform will get you the rest of the debugging experience (auto install, remote shell commands, and much much more.

Sounds promising! I'm not sure I totally understand the interplay of lldb-platform and gdbserver in this case --- on the remote debug side, is lldb attaching to two processes, an lldb-platform (for the get/put files, install executables, etc. platform-type operations) and then also the gdbserver (or similar) to do the remote target process control? Do I have that right? If so - what's the basic idea behind having those be two separate pieces? Thanks!

Yes, one process lldb-platform gets run and attached to. It can spawn a new GDB server for each process that it wants to debug. There is a packet that says "launch GDB server and hand me back the port number". Then using this port and the hostname of the remote platform, you can then attach to the remote GDB server and debug as many processes (attach or launch) as needed.

This is the biggest part that is missing right now for all many remote debugging scenarios: the ability to upload/install the needed files and run launch processes on the remote system.

Granted, the current lldb-platform is GDB remote packet based, but we can easily switch this to a better technology.

11

Ah, got it. Okay, thanks for explaining! And having it be two separate remote processes clearly makes sense given the lldb-platform remote part is managing the startup of the other, and has all the support needed to push new code.

There is probably some amount of overlap in lldb-platform and what we currently do with adb (Android Debug Bridge). Eventually I’ll need to figure out what makes sense to speed up the compile/deploy/debug/fix cycle.

12

Ah, got it. Okay, thanks for explaining! And having it be two separate remote processes clearly makes sense given the lldb-platform remote part is managing the startup of the other, and has all the support needed to push new code.

Yep.

There is probably some amount of overlap in lldb-platform and what we currently do with adb (Android Debug Bridge). Eventually I'll need to figure out what makes sense to speed up the compile/deploy/debug/fix cycle.

You can make a new platform named "remote-andriod" that can talk to adb, and just skip using the "lldb-platform" binary. Anything that is missing in that we need in the lldb_private::Platform class could then be added to adb, and if we are missing anything in the lldb_private::Platform compared to adb we could add to the platform API. How do you communicate with adb? Sockets?

13

>
> There is probably some amount of overlap in lldb-platform and what we
currently do with adb (Android Debug Bridge). Eventually I'll need to
figure out what makes sense to speed up the compile/deploy/debug/fix cycle.

You can make a new platform named "remote-andriod" that can talk to adb,
and just skip using the "lldb-platform" binary. Anything that is missing in
that we need in the lldb_private::Platform class could then be added to
adb, and if we are missing anything in the lldb_private::Platform compared
to adb we could add to the platform API. How do you communicate with adb?
Sockets?

ADB runs on the host/local side, and it knows how to forward ports to

devices (typically over USB). So, we talk to the local ADB port that then
forwards communication to the actual device. I'll need to have a look at
the guts of it - it might make sense to only use it as a port forwarder and
go with an lldb-platform on the device side and just stick with that. I'll
have a deeper look at that once I get Android gdbserver (or equivalent)
working.

14

Take a look at the patch I sent you. It uses an lldb platform based on ADB to set up the port forwarding and possibly copy files then connects to gdbserver. The implementation is rough, but the basic idea seemed to work pretty well. I was using it to copy over and launch a new version of gdbserver because the one that came with the x86 emulator at the time didn’t work.

-Andy

15

Great, thanks Andy. Right now I’m just at the point of using the stock adb/gdbserver and see what that looks like. I’ll definitely be looking at your patch in a bit here :slight_smile:

16

If you are going to use a stock gdbserver binary, you will need to make a target definition python file. We have a few examples checked in:

svn cat http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/x86_64_linux_target_definition.py
svn cat http://llvm.org/svn/llvm-project/lldb/trunk/examples/python/x86_64_target_definition.py

When debugging using GDB, you will first need to see the exact registers that the gdbserver supplies:

(gdb) maint print raw-registers

Then you will need to make a register definition file based off of that and point lldb to use it:

(lldb) settings set plugin.process.gdb-remote.target-definition-file /path/to/trunk/examples/python/x86_64_target_definition.py

Then LLDB will be able to debug to a remote gdb server that doesn't support any of the dynamic register definition packets.

Greg

17

Perfect, thanks!

Technically I’ll be using whatever the Android NDK gdbserver is - I’ll need to track down what (if any) patches are applied to that.

18

Todd,

I went ahead and created a new "lldb-gdbserver" tool in "trunk/tools/lldb-gdbserver" in the repository.

I also started the Host layer abstraction (see include/Host/Debug.h) for processes (NativeProcessProtocol) and threads (NativeThreadProtocol).

In order to get a lldb-gdbserver up and running, it will be a matter of adding new packet support to the GDBRemoteCommunicationServer class to accept all of the standard GDB remote packets used for debugging. The GDBRemoteCommunicationServer class currently supports the lldb-platform packets, but doesn't implement a lot of the normal GDB remote packets.

So the work flow to get this working on linux will be:
1 - Implement a linux version NativeProcessProtocol and NativeThreadProtocol by having them used the ProcessMonitor stuff that is down in the linux native debugger plug-in.
2 - Implement any needed GDB remote packets in GDBRemoteCommunicationServer and back them by a single instance of NativeProcessProtocol when launching or attaching to a process.

If you need any help let me know. I am sure there is stuff missing from NativeProcessProtocol and NativeThreadProtocol, so let me know if you need anything else. Also please ask questions as you go if you need any help.

Greg Clayton

19

You rock, Greg! Thanks :slight_smile:

20

it might make sense to only use it as a port forwarder and go with an lldb-platform on the device side and just stick with that. I’ll have a deeper look at that once I get Android gdbserver (or equivalent) working.

It’s very likely I’ll need to base lldb-platform entirely around adb now that I think about it. adb activity is more or less the security gatekeeper - if a user doesn’t want a device to be debugged, it can prevent adb from attaching to it. I doubt I’d be able to get a useful lldb-platform exe on the device with the right permissions for a non-rooted build (i.e. a standard device image). So an lldb-platform that builds around adb on Android is likely going to be the way for me to at least start. This may require changing adb in some cases depending on exactly what lldb-platform requires.

Andy - I’ll be looking at your patch starting today. I hit my first crash in lldb on connecting to a stock android gdbserver against a Nexus 10 yesterday doing nothing other than attempting to connect :slight_smile: