lldb on linux -- immediate crash upon attaching to process

I’m trying lldb on linux x86_64. Upon attaching to any process with “pro attach --pid XXX”, I get
an immediate core dump via referencing an invalid mutex.

I know linux is a new platform for lldb, and I’ve just recently started exploring lldb on linux so
so please forgive me if this is already a well known issue. (Is there a bug database to search?).

In case it helps, I attach a debugging log with strack trace and a small test case to make it
easy to reproduce. This was running on Ubuntu 10.04, with the latest lldb (svn r127570), and
llvm and clang at svn r124349, as Steve Wilson kindly indicated was necessary for lldb to build.

lldb was built with this configuration.

$ /home/jaten/pkg/latest-svn-llvm/llvm/configure --enable-optimized --enable-debug-runtime --enable-debug-symbols --enable-profiling --enable-jit --enable-doxygen --enable-pic --enable-shared --enable-assertions --enable-targets=x86_64 REQUIRES_RTTI=1

Let me know if I can be of further assistance in tracking this down. If it’s a known problem and
somebody can sketch a solution, I may be able to implement it.

thanks,
Jason

lldb-core-dump-on-process-attach-log.txt.gz (14 KB)

I'm trying lldb on linux x86_64. Upon attaching to any process with "pro
attach --pid XXX", I get
an immediate core dump via referencing an invalid mutex.

Heh, ya. Attach is simply not implemented. Until it is, I would
suggest "don't do that" :slight_smile:

I know linux is a new platform for lldb, and I've just recently started
exploring lldb on linux so
so please forgive me if this is already a well known issue. (Is there a bug
database to search?).

We have only the most basic infrastructure on linux. There is still a
significant amount of work that needs to be done before LLDB even begins
to approach "usable" on that platform.

In case it helps, I attach a debugging log with strack trace and a small
test case to make it
easy to reproduce. This was running on Ubuntu 10.04, with the latest lldb
(svn r127570), and
llvm and clang at svn r124349, as Steve Wilson kindly indicated was
necessary for lldb to build.

lldb was built with this configuration.

$ /home/jaten/pkg/latest-svn-llvm/llvm/configure --enable-optimized
--enable-debug-runtime --enable-debug-symbols --enable-profiling
--enable-jit --enable-doxygen --enable-pic --enable-shared
--enable-assertions --enable-targets=x86_64 REQUIRES_RTTI=1

Let me know if I can be of further assistance in tracking this down. If it's
a known problem and
somebody can sketch a solution, I may be able to implement it.

Well, in a nutshell you would need to implement something similar to
what ProcessLinux::DoLaunch does, but in this case you want things to
boil down to a ptrace(ATTACH) instead of a fork() + ptrace(TRACEME).

The basic sketch would be:

   - Define a new ProcessMonitor ctor that takes a pid as argument.

   - Define ProcessMonitor::Attach which does the actual ptrace magic.

   - Write a another StartOperationThread method that takes a (new)
     AttachArgs struct as argument (could just contain the pid for now)
     and sets up the monitoring business in essentially the same way as
     the current launch-based code does. Probably rename
     OperationThread to LaunchOpThread or similar and write your own
     AttachOpThread analog.

It would certainly be nice to have that implemented. I do not see
anything that would cause any complications off hand, and it should
remain fairly isolated from all the other work that needs to happen wrt
linux support.

Thanks Steve! I scoped out the work a little bit, mostly by stepping through in the debuggers both the Xcode version and the current Linux version. Btw it looks like the current version of lldb has been incremented (now r127600), which is very good news.

I note that the main contrast is this: the darwin built lldb uses the ProcessGDBRemote class, implemented in the “llvm/tools/lldb/source/Plugins/Process/gdb-remote” directory, rather than ProcessLinux.

The curious thing is: when I look through the gdb-remote code, there are only two lines that are #ifdef APPLE. It seems fairly reusable.

My naive question then is, why not just reuse the ProcessGDBRemote code for Linux as well? There’s probably higher level design issues that I’m not familiar with, so anyone on lldb-dev should feel free to chime in here. The second lazy inclination is to just port that code to linux if it must go in it’s own directory.

Let me know what you think. I’m probably asking silly questions, but I’m just trying to get my bearings. Please bear with me! :slight_smile:

Thanks,
Jason

p.s. the one thing that kept me from trying this directly was figuring out where in the Makefile system this got chosen, because by default on linux, the gdb-remote directory isn’t built. If anyone knows where this controlled, please point it out. Thank you!

Well, in a nutshell you would need to implement something similar to
what ProcessLinux::DoLaunch does, but in this case you want things to
boil down to a ptrace(ATTACH) instead of a fork() + ptrace(TRACEME).
The basic sketch would be:
  - Define a new ProcessMonitor ctor that takes a pid as argument.
  - Define ProcessMonitor::Attach which does the actual ptrace magic.
  - Write a another StartOperationThread method that takes a (new)
    AttachArgs struct as argument (could just contain the pid for now)
    and sets up the monitoring business in essentially the same way as
    the current launch-based code does. Probably rename
    OperationThread to LaunchOpThread or similar and write your own
    AttachOpThread analog.
It would certainly be nice to have that implemented. I do not see
anything that would cause any complications off hand, and it should
remain fairly isolated from all the other work that needs to happen wrt
linux support.

Thanks Steve! I scoped out the work a little bit, mostly by stepping through in the debuggers both the Xcode version and the current Linux version. Btw it looks like the current version of lldb has been incremented (now r127600), which is very good news.

Yes we recently updated to get needed disassembler fixes.

I note that the main contrast is this: the darwin built lldb uses the ProcessGDBRemote class, implemented in the "llvm/tools/lldb/source/Plugins/Process/gdb-remote" directory, rather than ProcessLinux.

The curious thing is: when I look through the gdb-remote code, there are only two lines that are #ifdef APPLE. It seems fairly reusable.

It is very reusable and can be used for just about any debugging. It can probably actually be used with the GDB produced gdbserver binary, but that would need to be modified to support some of the extra new packets we added to the GDB remote protocol for register set discovery (qRegisterInfo) and host info (qHostInfo). The register info packets allow complete discovery of process registers and include all of the DWARF and GCC register numberings, as well the generic registers ("pc" (rip/eip on x86), "sp" (rsp/esp on x86), "fp" (rbp/ebp on x86), "ra" (return address (n/a on x86), "flags" (rflags/eflags on x86). Example packets for an x86_64 register context on MacOSX looks like:

-> $qRegisterInfo0#00
<- $name:rax;bitsize:64;offset:0;encoding:uint;format:hex;set:General Purpose Registers;gcc:0;dwarf:0;#00
-> $qRegisterInfo1#00
<- $name:rbx;bitsize:64;offset:8;encoding:uint;format:hex;set:General Purpose Registers;gcc:3;dwarf:3;#00
-> $qRegisterInfo2#00
<- $name:rcx;bitsize:64;offset:16;encoding:uint;format:hex;set:General Purpose Registers;gcc:2;dwarf:2;#00
-> $qRegisterInfo3#00
<- $name:rdx;bitsize:64;offset:24;encoding:uint;format:hex;set:General Purpose Registers;gcc:1;dwarf:1;#00
-> $qRegisterInfo4#00
<- $name:rdi;bitsize:64;offset:32;encoding:uint;format:hex;set:General Purpose Registers;gcc:5;dwarf:5;#00
-> $qRegisterInfo5#00
<- $name:rsi;bitsize:64;offset:40;encoding:uint;format:hex;set:General Purpose Registers;gcc:4;dwarf:4;#00
-> $qRegisterInfo6#00
<- $name:rbp;alt-name:fp;bitsize:64;offset:48;encoding:uint;format:hex;set:General Purpose Registers;gcc:6;dwarf:6;generic:fp;#00
-> $qRegisterInfo7#00
<- $name:rsp;alt-name:sp;bitsize:64;offset:56;encoding:uint;format:hex;set:General Purpose Registers;gcc:7;dwarf:7;generic:sp;#00
-> $qRegisterInfo8#00
<- $name:r8;bitsize:64;offset:64;encoding:uint;format:hex;set:General Purpose Registers;gcc:8;dwarf:8;#00
-> $qRegisterInfo9#00
<- $name:r9;bitsize:64;offset:72;encoding:uint;format:hex;set:General Purpose Registers;gcc:9;dwarf:9;#00
-> $qRegisterInfoa#00
<- $name:r10;bitsize:64;offset:80;encoding:uint;format:hex;set:General Purpose Registers;gcc:10;dwarf:10;#00
-> $qRegisterInfob#00
<- $name:r11;bitsize:64;offset:88;encoding:uint;format:hex;set:General Purpose Registers;gcc:11;dwarf:11;#00
-> $qRegisterInfoc#00
<- $name:r12;bitsize:64;offset:96;encoding:uint;format:hex;set:General Purpose Registers;gcc:12;dwarf:12;#00
-> $qRegisterInfod#00
<- $name:r13;bitsize:64;offset:104;encoding:uint;format:hex;set:General Purpose Registers;gcc:13;dwarf:13;#00
-> $qRegisterInfoe#00
<- $name:r14;bitsize:64;offset:112;encoding:uint;format:hex;set:General Purpose Registers;gcc:14;dwarf:14;#00
-> $qRegisterInfof#00
<- $name:r15;bitsize:64;offset:120;encoding:uint;format:hex;set:General Purpose Registers;gcc:15;dwarf:15;#00
-> $qRegisterInfo10#00
<- $name:rip;alt-name:pc;bitsize:64;offset:128;encoding:uint;format:hex;set:General Purpose Registers;gcc:16;dwarf:16;generic:pc;#00
-> $qRegisterInfo11#00
<- $name:rflags;alt-name:flags;bitsize:64;offset:136;encoding:uint;format:hex;set:General Purpose Registers;#00
-> $qRegisterInfo12#00
<- $name:cs;bitsize:64;offset:144;encoding:uint;format:hex;set:General Purpose Registers;#00
-> $qRegisterInfo13#00
<- $name:fs;bitsize:64;offset:152;encoding:uint;format:hex;set:General Purpose Registers;#00
-> $qRegisterInfo14#00
<- $name:gs;bitsize:64;offset:160;encoding:uint;format:hex;set:General Purpose Registers;#00
-> $qRegisterInfo15#00
<- $name:fctrl;bitsize:16;offset:176;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo16#00
<- $name:fstat;bitsize:16;offset:178;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo17#00
<- $name:ftag;bitsize:8;offset:180;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo18#00
<- $name:fop;bitsize:16;offset:182;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo19#00
<- $name:fioff;bitsize:32;offset:184;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo1a#00
<- $name:fiseg;bitsize:16;offset:188;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo1b#00
<- $name:fooff;bitsize:32;offset:192;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo1c#00
<- $name:foseg;bitsize:16;offset:196;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo1d#00
<- $name:mxcsr;bitsize:32;offset:200;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo1e#00
<- $name:mxcsrmask;bitsize:32;offset:204;encoding:uint;format:hex;set:Floating Point Registers;#00
-> $qRegisterInfo1f#00
<- $name:stmm0;bitsize:80;offset:208;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:33;dwarf:33;#00
-> $qRegisterInfo20#00
<- $name:stmm1;bitsize:80;offset:224;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:34;dwarf:34;#00
-> $qRegisterInfo21#00
<- $name:stmm2;bitsize:80;offset:240;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:35;dwarf:35;#00
-> $qRegisterInfo22#00
<- $name:stmm3;bitsize:80;offset:256;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:36;dwarf:36;#00
-> $qRegisterInfo23#00
<- $name:stmm4;bitsize:80;offset:272;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:37;dwarf:37;#00
-> $qRegisterInfo24#00
<- $name:stmm5;bitsize:80;offset:288;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:38;dwarf:38;#00
-> $qRegisterInfo25#00
<- $name:stmm6;bitsize:80;offset:304;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:39;dwarf:39;#00
-> $qRegisterInfo26#00
<- $name:stmm7;bitsize:80;offset:320;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:40;dwarf:40;#00
-> $qRegisterInfo27#00
<- $name:xmm0;bitsize:128;offset:336;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:17;dwarf:17;#00
-> $qRegisterInfo28#00
<- $name:xmm1;bitsize:128;offset:352;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:18;dwarf:18;#00
-> $qRegisterInfo29#00
<- $name:xmm2;bitsize:128;offset:368;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:19;dwarf:19;#00
-> $qRegisterInfo2a#00
<- $name:xmm3;bitsize:128;offset:384;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:20;dwarf:20;#00
-> $qRegisterInfo2b#00
<- $name:xmm4;bitsize:128;offset:400;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:21;dwarf:21;#00
-> $qRegisterInfo2c#00
<- $name:xmm5;bitsize:128;offset:416;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:22;dwarf:22;#00
-> $qRegisterInfo2d#00
<- $name:xmm6;bitsize:128;offset:432;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:23;dwarf:23;#00
-> $qRegisterInfo2e#00
<- $name:xmm7;bitsize:128;offset:448;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:24;dwarf:24;#00
-> $qRegisterInfo2f#00
<- $name:xmm8;bitsize:128;offset:464;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:25;dwarf:25;#00
-> $qRegisterInfo30#00
<- $name:xmm9;bitsize:128;offset:480;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:26;dwarf:26;#00
-> $qRegisterInfo31#00
<- $name:xmm10;bitsize:128;offset:496;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:27;dwarf:27;#00
-> $qRegisterInfo32#00
<- $name:xmm11;bitsize:128;offset:512;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:28;dwarf:28;#00
-> $qRegisterInfo33#00
<- $name:xmm12;bitsize:128;offset:528;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:29;dwarf:29;#00
-> $qRegisterInfo34#00
<- $name:xmm13;bitsize:128;offset:544;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:30;dwarf:30;#00
-> $qRegisterInfo35#00
<- $name:xmm14;bitsize:128;offset:560;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:31;dwarf:31;#00
-> $qRegisterInfo36#00
<- $name:xmm15;bitsize:128;offset:576;encoding:vector;format:vector-uint8;set:Floating Point Registers;gcc:32;dwarf:32;#00
-> $qRegisterInfo37#00
<- $name:trapno;bitsize:32;offset:696;encoding:uint;format:hex;set:Exception State Registers;#00
-> $qRegisterInfo38#00
<- $name:err;bitsize:32;offset:700;encoding:uint;format:hex;set:Exception State Registers;#00
-> $qRegisterInfo39#00
<- $name:faultvaddr;bitsize:64;offset:704;encoding:uint;format:hex;set:Exception State Registers;#00
-> $qRegisterInfo3a#00
<- $E45#00

The host info gives us the target triple information. On apple it looks like:

-> $qHostInfo#00
<- $cputype:16777223;cpusubtype:3;ostype:Darwin;vendor:apple;endian:little;ptrsize:8;#00

We have also added a few packets that allow for memory allocation and deallocation with read/write/execute permissions. So the GDB remote can be used pretty easily with a few additions to many pre-existing GDB server source bases.

My naive question then is, why not just reuse the ProcessGDBRemote code for Linux as well?

There is something to be said about native debugging and getting the fastest debugging speeds. Debugging using the remote protocol slows things down a little bit, but not too much, so that would be a major reason to be opposed to using the remote protocol.

The nice thing about using process GDB Remnote and the reason we use it on MacOSX, is that it gets us remote debugging almost for free. Very little changes need to happen since we just connect to a different ip and port number.

There's probably higher level design issues that I'm not familiar with, so anyone on lldb-dev should feel free to chime in here. The second lazy inclination is to just port that code to linux if it must go in it's own directory.

The source code from "debugserver" is a mix of old code and some new code. I would like to re-write a lot of it to take advantage of many of the classes that exist inside LLDB (StringStream, Communication, Connectiond and GDBRemoteCommunication to name a few). It was written long before LLDB was around and has been slowly modernized, though it remains crufty. It is currently tailored to MacOSX only, but that being said, a new source base that re-uses the lldb_private communication classes. GDBRemoteCommunication.cpp/h can be reused in a newer binary very easily and adapted to be the server side instead of client side with a few modifications.

That being said, there is a nice abstraction that can be used with debugserver, check out the DNB.h file. This is designed to be a C interface to a native debugger on a native host. I had adapted it to do pretty much all we need to do for MacOSX and the current plan is to eventually re-use this DNB.h/DNB.cpp (the macosx version) in ProcessMacOSX. So we might be able to do the same for linux. Steps would involve:

1 - Modifying DNB.h to make sure it does all we need it to for all platforms
2 - Grab the code from within ProcessLinux and put it in a new DNB.cpp implementation for linux
3 - Make the ProcessLinux code us the DNB.h interface
4 - Reuse DNB.cpp/.h for linux in a new or modified version of "debugserver"

This way we maintain one codebase for native debugging, and then can leverage that code between the native and GDB remote debugserver binaries.

Let me know what you think. I'm probably asking silly questions, but I'm just trying to get my bearings. Please bear with me! :slight_smile:

Everyone, let me know what you think about the above proposal. The nice thing too is that once debugging for a native platform is abtracted in DNB.h (yes we can rename the header file, this used to stand for "debug nub"), then others can download the LLDB source code any can be able to run a native debug session without needing to have any of the LLDB engine overhead making it very portable.

Thanks,
Jason

p.s. the one thing that kept me from trying this directly was figuring out where in the Makefile system this got chosen, because by default on linux, the gdb-remote directory isn't built. If anyone knows where this controlled, please point it out. Thank you!

I will defer to Stephen Wilson on this one.

Greg Clayton

1 - Modifying DNB.h to make sure it does all we need it to for all platforms
2 - Grab the code from within ProcessLinux and put it in a new DNB.cpp implementation for linux
3 - Make the ProcessLinux code us the DNB.h interface
4 - Reuse DNB.cpp/.h for linux in a new or modified version of "debugserver"

At first I was into this idea. If we can keep DNB.h C compatible then I
see lots of benefits to the transition. I am particularly thinking
about the embedded developer here who may not have a C++ runtime but who
might like a simple way to hook their tools into LLDB so that they can
run on their boards, workstations, emulators, etc. Even more generally,
I think giving LLDB a window into plugins/tools that can provide a C API
is a good thing.

However, I have to admit to being a little concerned about having two
very similar API's side by side. I am quite tempted to suggest that
when/if the current debug server gets an overhaul it be rewritten to use
the Process API and use similar idioms of "plugin resource discovery" to
provide a generic/portable implementation that does a
packet-to-process-plugin translation thing.

I do not think that would necessarily leave C programmers out in the
cold: they would still be able to define Process plugin wrappers or
implement the gdb remote protocol.

So perhaps DNB.h might be best thought of *exclusively* as a C-level
API. If we wanted to support such a thing, then we could provide a
standard DNBProcess wrapper that C code could link against. Similarly
we could provide a C-level debugserver. In principle a "C-Bridge"
sub-project could be folded into LLDB pretty easily -- just like we do
for different OS platforms.

But perhaps I am thinking about things the wrong way, or I underestimate
the amount of work needed to "upgrade" the current debug server to use
the Process plugin framework... I am certainly still open to
either/different approaches.

> p.s. the one thing that kept me from trying this directly was
> figuring out where in the Makefile system this got chosen, because
> by default on linux, the gdb-remote directory isn't built. If
> anyone knows where this controlled, please point it out. Thank you!

I will defer to Stephen Wilson on this one.

We can't commit this yet, but here is the needed patch:

diff --git a/source/Plugins/Makefile b/source/Plugins/Makefile
index fd86444..43f0818 100644
--- a/source/Plugins/Makefile
+++ b/source/Plugins/Makefile
@@ -15,12 +15,11 @@ include $(LLDB_LEVEL)/../../Makefile.config
DIRS := ABI/MacOSX-i386 ABI/SysV-x86_64 Disassembler/llvm \
   ObjectContainer/BSD-Archive ObjectFile/ELF SymbolFile/DWARF \
   SymbolFile/Symtab Process/Utility DynamicLoader/Static \
- Platform
+ Platform Process/gdb-remote

ifeq ($(HOST_OS),Darwin)
DIRS += DynamicLoader/MacOSX-DYLD ObjectContainer/Universal-Mach-O \
- ObjectFile/Mach-O Process/gdb-remote \
- SymbolVendor/MacOSX Process/MacOSX-User
+ ObjectFile/Mach-O SymbolVendor/MacOSX Process/MacOSX-User
endif

ifeq ($(HOST_OS),Linux)

Jim Ingham would like to see us have debugserver re-use LLDB shared library (LLDB.framework on darwin, lldb.so on linux), and then debugserver would just use the native debugger plugin (ProcessMacOSX for darwin, ProcessLinux on linux) and avoid doing any symbol queries. We might need some sort of minimal mode within the LLDB to make sure we don't have both lldb and debugserver loading too many symbols, though with the way we have architected LLDB to only do lookups when asked, I think it would work just fine without any modifications.

I might try out this approach to see if it works. If it does, this means everyone should get remote debugging for free, and need only implement a single local Process plug-in.

So the executable shared library hierarchy would look like:

lldb.so
>-Process
> >- ProcessMacOSX
> >- ProcessLinux
>-...

Then the binaries would link to lldb.so:

lldb
>- lldb.so
    >- Process
       >- ProcessMacOSX
       >- ProcessLinux

debugserver
>- lldb.so
    >- Process
       >- ProcessMacOSX
       >- ProcessLinux

Comments?

Greg Clayton

Sounds reasonable to me, but I’m not really familiar enough with the architecture to have a super informed opinion. Following the Keep It Simple Silly (KISS), and You’re Not Going to Need It (YNGNI) principles certainly appeal to me.

I am however qualified and happy to provide testing and feedback as far as on the Linux side implementations go, and to do limited ports to linux of code originally implemented on darwin. Let me know if I can be of assistance in either of those two modes.

Jason

(Sorry about the delay -- it has been a busy week!)

Jim Ingham would like to see us have debugserver re-use LLDB shared
library (LLDB.framework on darwin, lldb.so on linux), and then
debugserver would just use the native debugger plugin (ProcessMacOSX
for darwin, ProcessLinux on linux) and avoid doing any symbol queries.
We might need some sort of minimal mode within the LLDB to make sure
we don't have both lldb and debugserver loading too many symbols,
though with the way we have architected LLDB to only do lookups when
asked, I think it would work just fine without any modifications.

I agree. Perhaps the ultimate aim should be to ensure that debugserver
never does any symbol lookups at all (since the client will sometimes be
the only one with access to the debug info).

In the absolute worst case we could add an LLDB specific set of
callbacks that the target (i.e. Process) can request over the remote
protocol. We could model that programmatically using a new "Environment"
class or similar passed as an argument to Process::CreateInstance that
would do the right thing depending on if we are running locally under
lldb or remotely under debugserver.

I might try out this approach to see if it works. If it does, this
means everyone should get remote debugging for free, and need only
implement a single local Process plug-in.

The only other thing I can think of immediately is that we would also
need a way to access the "target" file system state -- not just the
process state. IOW, an interface that translates into Host I/O packets
when doing remote debugging and native FS operations otherwise.

For example, in the linux dynamic loader we use /proc/pid/auxv which is
a kernel-provided array of values. We can pick them off the stack using
Process memory reads and writes but it is so much easier to just parse
the file contents. There are other examples of in-kernel trace and
debugging hooks that are provided thru different virtual file system
interfaces.

Perhaps what we need here is a TargetIO class with a standard set of
virtual methods for reading and writing files. The Process plugin could
provide a factory method that yields a reference to an instance of
TargetIO. The default behaviour would be to return a generic instance
that just does local open/read/write calls. The gdb-remote process
plugin would provide it's own specialization that provides an
implementation based on Host I/O packets.

Does any of the above sound sane/useful? If so I can start looking at
that side of things.