remote debugging from another platform

Hi,

Can LLDB do remote debugging from another platform? (Linux -> OSX for example)? If so which side would load the symbols/have to have the source files?

Is there documentation on how this protocol works?

Thanks,

Carlo Kok

Yes it can, but work is definitely needed. Linux to OSX will work because we have the lldb modified "debugserver" binary which can debug macosx apps. There is no linux equivalent because the current "debugserver" binary was initially made for the Mac only and it isn't well orgnanized to be used as a starting point for porting to other systems.

What I would like see happen, is in our Host layer (in "lldb/source/Host" and "lldb/include/lldb/Host"), we should have a native debug API that is very closely based on "lldb/tools/debugserver/source/DNB.h" and name it something like "lldb_private::NativeDebug". Once we do this, we can then support native debugging on every platform by making a new Process plugin in "lldb/source/Plugins/Process" called "Host" which would link against the new API in the lldb_private::NativeDebug. We then also have classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationServer.cpp

The "GDBRemoteCommunicationClient" class communicates with remote GDB servers.

The "GDBRemoteCommunicationServer" class implements the GDB server side of the connection. This is not currently being used in "debugserver". This "GDBRemoteCommunicationServer" class can then just link against the "lldb_private::NativeDebug" class to implement remote debugging. This way, both native and remote debugging always share the same base code.

There will need to be some work done on the build as well to make sure that all plug-ins needed for remote debugging get linked in. Right now there is code that disables many plug-ins when they shouldn't be disabled in "lldb/source/lldb.cpp" by using proprocessor directives.

As for trying out remote debugging from linux, the flow would be to first launch debugserver on the mac side:

darwin% debugserver linux.foo.com:12345 -- /bin/ls -lAF

Then launch lldb and connect on linux:

linux% lldb
(lldb) platform select --sysroot /remote/path/to/macosx/root remote-macosx
(lldb) file --arch x86_64 /remote/path/to/macosx/root/bin/ls
(lldb) process connect connect://darwin.foo.com:12345
...

We also have code in LLDB that manages platforms. We see above we switched platforms from the default "localhost" platform to the "remote-macosx" platform above. When we get platforms working, the flow will change to this:

darwin% lldb-platform linux.foo.com:12345

Now we have a platform on our macosx machine that can answer many things:

linux% lldb
(lldb) platform select remote-macosx
(lldb) platform connect connect://darwin.foo.com:12345

(lldb) platform process list
PID PARENT USER ARCH NAME
====== ====== ========== ======= ============================
16773 164 gclayton 6 x86_64-apple-macosx mdworker
16712 16711 gclayton 6 x86_64-apple-macosx tcsh
16709 164 gclayton 6 x86_64-apple-macosx Terminal
16705 164 gclayton 6 x86_64-apple-macosx mdworker
16668 1 gclayton 6 x86_64-apple-macosx XType.FontHelper
16652 1 gclayton 4 i386-apple-macosx qtkittrustedmoviesservice
16651 1 gclayton 4 i386-apple-macosx com.apple.qtkitserver
16649 164 gclayton 6 x86_64-apple-macosx WebKitPluginAgent
...

(lldb) platform process info 16712
Process information for process 16712:
    pid = 16712
parent = 16711
   name = tcsh
   file = /bin/tcsh
arg[0] = -tcsh
   arch = x86_64-apple-macosx
    uid = 501 (gclayton)
...

Now we have an actual connection to a remote platform and we can ask the platform launch a process for us:

(lldb) platform process launch -- /bin/ls -lAF

The platforms will be able to upload/download files, install new applications, locate remote copies of platform files on the current computer, and much more. We havea lot of that work started in our platform branch:

svn co http://llvm.org/svn/llvm-project/lldb/branches/lldb-platform-work lldb-platform

We have had time to finish it all up, but we did get started. The Xcode project for the MacOSX has a "lldb-platform" binary started that uses the GDB remote protocol to implement the communication between computers. It links against the "GDBRemoteCommunicationServer" and it uses this class to do most of the work. Take a look at:

tools/lldb-platform/lldb-platform.cpp

This will show you how easy it will be to implement the GDB remote communication for both the platform, and, in the future, for the lldb-debugserver that will wrap up the "lldb_private::NativeDebug" layer for remote debugging.

Greg Clayton

Op 27-8-2012 19:16, Greg Clayton schreef:

Yes it can, but work is definitely needed. Linux to OSX will work
because we have the lldb modified "debugserver" binary which can
debug macosx apps. There is no linux equivalent because the current
"debugserver" binary was initially made for the Mac only and it isn't
well orgnanized to be used as a starting point for porting to other
systems.

What I would like see happen, is in our Host layer (in
"lldb/source/Host" and "lldb/include/lldb/Host"), we should have a
native debug API that is very closely based on
"lldb/tools/debugserver/source/DNB.h" and name it something like
"lldb_private::NativeDebug". Once we do this, we can then support
native debugging on every platform by making a new Process plugin in
"lldb/source/Plugins/Process" called "Host" which would link against
the new API in the lldb_private::NativeDebug. We then also have
classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

Are there any technical limitations that keep these from working on windows (the client side of things, not the host)? (I tried to import it but there are a lot of missing include files which i presume is just because nobody ever did this and it's just missing defines).

We have a host layer in our code that is designed to abstract you from the OS your are running on, though I am sure it is far from perfect. One of the big challenges in making a windows port will be correctly implementing an abstraction layer that makes sense.

Does anyone else have any experience and comments for Windows? I seem to remember a few folks working on windows over that past year.

Greg Clayton

Op 4-9-2012 16:46, Greg Clayton schreef:

Op 27-8-2012 19:16, Greg Clayton schreef:

Yes it can, but work is definitely needed. Linux to OSX will work
because we have the lldb modified "debugserver" binary which can
debug macosx apps. There is no linux equivalent because the current
"debugserver" binary was initially made for the Mac only and it isn't
well orgnanized to be used as a starting point for porting to other
systems.

What I would like see happen, is in our Host layer (in
"lldb/source/Host" and "lldb/include/lldb/Host"), we should have a
native debug API that is very closely based on
"lldb/tools/debugserver/source/DNB.h" and name it something like
"lldb_private::NativeDebug". Once we do this, we can then support
native debugging on every platform by making a new Process plugin in
"lldb/source/Plugins/Process" called "Host" which would link against
the new API in the lldb_private::NativeDebug. We then also have
classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

Are there any technical limitations that keep these from working on windows (the client side of things, not the host)? (I tried to import it but there are a lot of missing include files which i presume is just because nobody ever did this and it's just missing defines).

We have a host layer in our code that is designed to abstract you from the OS your are running on, though I am sure it is far from perfect. One of the big challenges in making a windows port will be correctly implementing an abstraction layer that makes sense.

Does anyone else have any experience and comments for Windows? I seem to remember a few folks working on windows over that past year.

Greg Clayton

There are quite a few places that presume unix (like spawn.h, pthread.h); the patch attached has some small tweaks for this but is far from complete. Atm I'm stuck with regex.h (completely missing on vc++) and a read/write lock primitive (readwritelock.h wants it). Win32 doesn't seem to have an equivalent.

lldb.patch (17.8 KB)

Op 4-9-2012 21:35, Carlo Kok schreef:

Op 4-9-2012 16:46, Greg Clayton schreef:

Op 27-8-2012 19:16, Greg Clayton schreef:

Yes it can, but work is definitely needed. Linux to OSX will work
because we have the lldb modified "debugserver" binary which can
debug macosx apps. There is no linux equivalent because the current
"debugserver" binary was initially made for the Mac only and it isn't
well orgnanized to be used as a starting point for porting to other
systems.

What I would like see happen, is in our Host layer (in
"lldb/source/Host" and "lldb/include/lldb/Host"), we should have a
native debug API that is very closely based on
"lldb/tools/debugserver/source/DNB.h" and name it something like
"lldb_private::NativeDebug". Once we do this, we can then support
native debugging on every platform by making a new Process plugin in
"lldb/source/Plugins/Process" called "Host" which would link against
the new API in the lldb_private::NativeDebug. We then also have
classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

Are there any technical limitations that keep these from working on
windows (the client side of things, not the host)? (I tried to import
it but there are a lot of missing include files which i presume is
just because nobody ever did this and it's just missing defines).

We have a host layer in our code that is designed to abstract you from
the OS your are running on, though I am sure it is far from perfect.
One of the big challenges in making a windows port will be correctly
implementing an abstraction layer that makes sense.

Does anyone else have any experience and comments for Windows? I seem
to remember a few folks working on windows over that past year.

Greg Clayton

There are quite a few places that presume unix (like spawn.h,
pthread.h); the patch attached has some small tweaks for this but is far
from complete. Atm I'm stuck with regex.h (completely missing on vc++)
and a read/write lock primitive (readwritelock.h wants it). Win32
doesn't seem to have an equivalent.

Hmm rest got cut off.

What I wanted to ask here is, is this the right way to go about this? (it's not a proper patch).

Op 4-9-2012 21:35, Carlo Kok schreef:

Op 4-9-2012 16:46, Greg Clayton schreef:

Op 27-8-2012 19:16, Greg Clayton schreef:

Yes it can, but work is definitely needed. Linux to OSX will work
because we have the lldb modified "debugserver" binary which can
debug macosx apps. There is no linux equivalent because the current
"debugserver" binary was initially made for the Mac only and it isn't
well orgnanized to be used as a starting point for porting to other
systems.

What I would like see happen, is in our Host layer (in
"lldb/source/Host" and "lldb/include/lldb/Host"), we should have a
native debug API that is very closely based on
"lldb/tools/debugserver/source/DNB.h" and name it something like
"lldb_private::NativeDebug". Once we do this, we can then support
native debugging on every platform by making a new Process plugin in
"lldb/source/Plugins/Process" called "Host" which would link against
the new API in the lldb_private::NativeDebug. We then also have
classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

Are there any technical limitations that keep these from working on
windows (the client side of things, not the host)? (I tried to import
it but there are a lot of missing include files which i presume is
just because nobody ever did this and it's just missing defines).

We have a host layer in our code that is designed to abstract you from
the OS your are running on, though I am sure it is far from perfect.
One of the big challenges in making a windows port will be correctly
implementing an abstraction layer that makes sense.

Does anyone else have any experience and comments for Windows? I seem
to remember a few folks working on windows over that past year.

Greg Clayton

There are quite a few places that presume unix (like spawn.h,
pthread.h); the patch attached has some small tweaks for this but is far
from complete. Atm I'm stuck with regex.h (completely missing on vc++)
and a read/write lock primitive (readwritelock.h wants it). Win32
doesn't seem to have an equivalent.

Hmm rest got cut off.

What I wanted to ask here is, is this the right way to go about this? (it's not a proper patch).

There are two ways that I see:
1 - completely hide everything in the host layer, no unix headers anywhere that isn't in the host code.
2 - require that a posix compliant unix layer is needed. I believe windows has cygwin and a a few other *nix variants that can be installed.

As far as the regular expression stuff goes, I believe llvm/clang have a regex implementation built in. We could switch over to using that.

Greg Clayton

Op 4-9-2012 21:43, Greg Clayton schreef:

Op 4-9-2012 21:35, Carlo Kok schreef:

Op 4-9-2012 16:46, Greg Clayton schreef:

Op 27-8-2012 19:16, Greg Clayton schreef:

Yes it can, but work is definitely needed. Linux to OSX will work
because we have the lldb modified "debugserver" binary which can
debug macosx apps. There is no linux equivalent because the current
"debugserver" binary was initially made for the Mac only and it isn't
well orgnanized to be used as a starting point for porting to other
systems.

What I would like see happen, is in our Host layer (in
"lldb/source/Host" and "lldb/include/lldb/Host"), we should have a
native debug API that is very closely based on
"lldb/tools/debugserver/source/DNB.h" and name it something like
"lldb_private::NativeDebug". Once we do this, we can then support
native debugging on every platform by making a new Process plugin in
"lldb/source/Plugins/Process" called "Host" which would link against
the new API in the lldb_private::NativeDebug. We then also have
classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

Are there any technical limitations that keep these from working on
windows (the client side of things, not the host)? (I tried to import
it but there are a lot of missing include files which i presume is
just because nobody ever did this and it's just missing defines).

We have a host layer in our code that is designed to abstract you from
the OS your are running on, though I am sure it is far from perfect.
One of the big challenges in making a windows port will be correctly
implementing an abstraction layer that makes sense.

Does anyone else have any experience and comments for Windows? I seem
to remember a few folks working on windows over that past year.

Greg Clayton

There are quite a few places that presume unix (like spawn.h,
pthread.h); the patch attached has some small tweaks for this but is far
from complete. Atm I'm stuck with regex.h (completely missing on vc++)
and a read/write lock primitive (readwritelock.h wants it). Win32
doesn't seem to have an equivalent.

Hmm rest got cut off.

What I wanted to ask here is, is this the right way to go about this? (it's not a proper patch).

There are two ways that I see:
1 - completely hide everything in the host layer, no unix headers anywhere that isn't in the host code.
2 - require that a posix compliant unix layer is needed. I believe windows has cygwin and a a few other *nix variants that can be installed.

As far as the regular expression stuff goes, I believe llvm/clang have a regex implementation built in. We could switch over to using that.

If at all possible, avoiding the posix later (cygwin) would be good; llvm/clang don't need it either. Esp as cygwin does not work well with VC++.

Op 4-9-2012 21:46, Carlo Kok schreef:

Op 4-9-2012 21:43, Greg Clayton schreef:

Op 4-9-2012 21:35, Carlo Kok schreef:

Op 4-9-2012 16:46, Greg Clayton schreef:

Op 27-8-2012 19:16, Greg Clayton schreef:

Yes it can, but work is definitely needed. Linux to OSX will work
because we have the lldb modified "debugserver" binary which can
debug macosx apps. There is no linux equivalent because the current
"debugserver" binary was initially made for the Mac only and it
isn't
well orgnanized to be used as a starting point for porting to other
systems.

What I would like see happen, is in our Host layer (in
"lldb/source/Host" and "lldb/include/lldb/Host"), we should have a
native debug API that is very closely based on
"lldb/tools/debugserver/source/DNB.h" and name it something like
"lldb_private::NativeDebug". Once we do this, we can then support
native debugging on every platform by making a new Process plugin in
"lldb/source/Plugins/Process" called "Host" which would link against
the new API in the lldb_private::NativeDebug. We then also have
classes which implement remote GDB debugging inside lldb:

lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
lldb/source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp

Are there any technical limitations that keep these from working on
windows (the client side of things, not the host)? (I tried to import
it but there are a lot of missing include files which i presume is
just because nobody ever did this and it's just missing defines).

We have a host layer in our code that is designed to abstract you from
the OS your are running on, though I am sure it is far from perfect.
One of the big challenges in making a windows port will be correctly
implementing an abstraction layer that makes sense.

Does anyone else have any experience and comments for Windows? I seem
to remember a few folks working on windows over that past year.

Greg Clayton

There are quite a few places that presume unix (like spawn.h,
pthread.h); the patch attached has some small tweaks for this but is
far
from complete. Atm I'm stuck with regex.h (completely missing on vc++)
and a read/write lock primitive (readwritelock.h wants it). Win32
doesn't seem to have an equivalent.

Hmm rest got cut off.

What I wanted to ask here is, is this the right way to go about this?
(it's not a proper patch).

There are two ways that I see:
1 - completely hide everything in the host layer, no unix headers
anywhere that isn't in the host code.
2 - require that a posix compliant unix layer is needed. I believe
windows has cygwin and a a few other *nix variants that can be installed.

As far as the regular expression stuff goes, I believe llvm/clang have
a regex implementation built in. We could switch over to using that.

If at all possible, avoiding the posix later (cygwin) would be good;
llvm/clang don't need it either. Esp as cygwin does not work well with
VC++.

I continued with this to see how far I can get:
With a slight tweak in llvm (to get back the original string from a regex class), and a few tweaks over lldb itself the compile progress gets pretty far (doesn't compile as is with the patch yet, some of the errors I get on Windows stop me from seeing all errors)

SymbolVendorMacOSX.cpp has a dependency on libxml & AvailabilityMacros.h
process.cpp has a lot of platform dependent stuff
platform.cpp depends on getops.h, not sure if this works on windows too but it's not standard

replaced the regex with llvms' regex (Requiers a tiny patch in llvm itself)

GDBRemoteCommunicationServer.cpp has a lot of platform dependent stuff
objectcontainerbsdarchive.cpp depends on ar.h (which is missing)

For emulationstatearm.h:
Error 4241 error C2380: type(s) preceding 'sd_regs' (constructor with return type, or illegal redefinition of current class-name?) lldb\source\plugins\instruction\arm\emulationstatearm.h 90 1 lldb
which causes like 100 follow-up errors.

DisassemblerLLVMC.cpp depends on regex.h (c style); probably can be ported to llvm regex or llvm's internal_regex stuff, which seem to match in the apis.

Abstracting everything in a windows/ specific folder does not look like it's very practical, __attribute__ isn't even supported in VC++ and used all over the place.

That said, I think I can get this all working so at the very least remote debugging from windows to other platforms works, maybe with some help on the issues above.

llvm_patch.zip (16.8 KB)

Carlo: I created a branch for the windows work to happen in:

svn co http://llvm.org/svn/llvm-project/lldb/branches/windows

You can check in anything you want into here. We currently lock down the build to a specific version of llvm/clang which is mentioned in the "$llvm_revision" and "$clang_revision" variable in the "lldb/scripts/build-llvm.pl" script. It is currently set to: r152265 for both. You have patches that need to be applied to the sources then you should include those patches in "lldb/scripts/llvm.amalgamated.diff" and the "lldb/scripts/clang.amalgamated.diff". These will then be applied to the llvm/clang builds before we start building LLDB.

I continued with this to see how far I can get:
With a slight tweak in llvm (to get back the original string from a regex class), and a few tweaks over lldb itself the compile progress gets pretty far (doesn't compile as is with the patch yet, some of the errors I get on Windows stop me from seeing all errors)

SymbolVendorMacOSX.cpp has a dependency on libxml & AvailabilityMacros.h

Feel free to #ifdef around this for now with an appropriate "TODO: cross build issues" comment

process.cpp has a lot of platform dependent stuff

platform.cpp depends on getops.h, not sure if this works on windows too but it's not standard

The includes for Platform.cpp have:

#include "lldb/Target/Platform.h"
#include "lldb/Breakpoint/BreakpointIDList.h"
#include "lldb/Core/Error.h"
#include "lldb/Core/Log.h"
#include "lldb/Core/ModuleSpec.h"
#include "lldb/Core/PluginManager.h"
#include "lldb/Host/FileSpec.h"
#include "lldb/Host/Host.h"
#include "lldb/Target/Process.h"
#include "lldb/Target/Target.h"

It is probably in one of these .h files that tries to #include the "getops.h", you will need to locate that file and work around it somehow.

replaced the regex with llvms' regex (Requiers a tiny patch in llvm itself)

Sounds good, please add the appropriate patch to the "lldb/scripts/llvm.amalgamated.diff" or "lldb/scripts/clang.amalgamated.diff" as needed.

GDBRemoteCommunicationServer.cpp has a lot of platform dependent stuff

The #include files are all from LLDB sources, again, check which #include inside these files needs to be fixed.

objectcontainerbsdarchive.cpp depends on ar.h (which is missing)

See if LLVM has the defines from ar.h abstracted into one of their header files. If they don't we will need to add them. Let me know what is missing (comment out the #include <ar.h> and let me know what errors you run into. Typically, LLVM and Clang usually have these defined with different names inside an appropriate namespace. We need to move away from depending on system header files.

For emulationstatearm.h:
Error 4241 error C2380: type(s) preceding 'sd_regs' (constructor with return type, or illegal redefinition of current class-name?) lldb\source\plugins\instruction\arm\emulationstatearm.h 90 1 lldb
which causes like 100 follow-up errors.

DisassemblerLLVMC.cpp depends on regex.h (c style); probably can be ported to llvm regex or llvm's internal_regex stuff, which seem to match in the apis.

Yes, no one should be using "regex.h" directly. What we really need to do is to have our lldb_private::RegularExpression class use the LLVM implementation and remove all #includes to "regex.h" so we abstract all regular expression stuff into one place.

Abstracting everything in a windows/ specific folder does not look like it's very practical, __attribute__ isn't even supported in VC++ and used all over the place.

For LLDB we need to make sure all __attribute__ things are abstracted into preprocess #define macros. It looks like this is primarily used for PRINTF string errors/warnings like

    void
    Error (const char *fmt, ...) __attribute__ ((format (printf, 2, 3)));

We can add a macro to "lldb/include/lldb/lldb-defines.h" that checks for windows (using a correct preprocessor test, not the one I use below):

#if defined(windows)
#define LLDB_PRINTF_FORMAT_ATTRIBUTE(n,m)
#else
#define LLDB_PRINTF_FORMAT_ATTRIBUTE(n,m) __attribute__ ((format (printf,n,m)))
#endif

Then the code above would become:

    void
    Error (const char *fmt, ...) LLDB_PRINTF_FORMAT_ATTRIBUTE(2,3);

That said, I think I can get this all working so at the very least remote debugging from windows to other platforms works, maybe with some help on the issues above.

Sounds good, again, checkout the new windows branch:

  http://llvm.org/svn/llvm-project/lldb/branches/windows

and make all your changes on that branch. When we have something we are happy with, we will merge this back into mainline.

Greg Clayton