debug server help

I'm having trouble using the LLDB apis to connect to a remote (OSX) debugserver. It connects, starts the process on the server, then it fails on this command in Process.cpp:
process_sp = Attach (attach_info, debugger, target, listener, error);

It tries to do a "qLaunchGDBServer" which returns an error.

Transcript of the traffic to/from the server:
http://pastebin.com/ZH6FUccY

I'm running debugserver on OSX using:

/Applications/Xcode45-DP1.app//Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/debugserver -g -v 0.0.0.0:3333

Using the LLDB apis I'm calling:

m_arch = new ArchSpec("x86_64-apple-macosx");
m_debugger = Debugger::CreateInstance(DebuggerLog, this);
m_platform = PlatformSP(PlatformRemoteGDBServer::CreateInstance(true, m_arch));
m_debugger->GetPlatformList().Append(m_platform, true);
m_listener = ListenerSP(new Listener("<list>"));
m_platform->ConnectRemote(Args("connect://192.168.178.2:3333"));

ProcessLaunchInfo li;
li.SetWorkingDirectory("/Users/ck");
li.SetExecutableFile(FileSpec("/Users/ck/a.out", false), true);
m_error.Clear();
m_process = m_platform->DebugProcess(li, *m_debugger.get(), m_target.get(), *m_listener.get(), m_error);

What am I missing?

Hi Carlo,

Before trying to diagnose, could you update to a more recent Xcode version? 4.5 DP1 is older than 4.4, IIRC.
And I remember having some bugs with the earlier DP*.

Are the debugserver and the framework client using the same lldb version?

Regards,

  Filipe

Op 14-9-2012 11:24, Filipe Cabecinhas schreef:

Hi Carlo,

Before trying to diagnose, could you update to a more recent Xcode version? 4.5 DP1 is older than 4.4, IIRC.
And I remember having some bugs with the earlier DP*.

Same for 4.4 (says debugserver-192)

Are the debugserver and the framework client using the same lldb version?

I have no good way of knowing what debugserver-192 is, but I doubt it; I use trunk/Windows as I'm trying to control this from Windows.

Op 14-9-2012 11:54, Carlo Kok schreef:

Op 14-9-2012 11:24, Filipe Cabecinhas schreef:

Hi Carlo,

Before trying to diagnose, could you update to a more recent Xcode
version? 4.5 DP1 is older than 4.4, IIRC.
And I remember having some bugs with the earlier DP*.

Same for 4.4 (says debugserver-192)

Are the debugserver and the framework client using the same lldb version?

I have no good way of knowing what debugserver-192 is, but I doubt it; I
use trunk/Windows as I'm trying to control this from Windows.

With the hidden logging enabled it goes pretty well until:
76 +0.000020 sec [0b53/1903]: got a timeout, continuing...
77 +0.000020 sec [0b53/1903]: ::mach_msg ( msg->{bits = 0, size = 0 remote_port = 0, local_port = 0, reserved = 0x0, id = 0x0}, option = 0x402, send_size = 0, rcv_size = 1024, rcv_name = 0x1803, timeout = 0, notify = 0)
Got a connection, waiting for debugger instructions.
78 +0.000072 sec [0b53/0707]: putpkt: $QCb55#00
79 +0.000027 sec [0b53/0707]: MachProcess::GetAvailableSTDOUT (&0x7fff5184ce40[256]) ...
80 +7.210365 sec [0b53/1603]: getpkt: $qLaunchGDBServer#00
81 +0.000182 sec [0b53/0707]: unimplemented packet: 'qLaunchGDBServer'

Looking at the lldb svn history qLaunchGDBServer has been around for more than a year. Am I missing something or is this just something that won't work?

Op 14-9-2012 12:38, Carlo Kok schreef:

Op 14-9-2012 11:54, Carlo Kok schreef:

Op 14-9-2012 11:24, Filipe Cabecinhas schreef:

Hi Carlo,

Before trying to diagnose, could you update to a more recent Xcode
version? 4.5 DP1 is older than 4.4, IIRC.
And I remember having some bugs with the earlier DP*.

Even latest lldb does 't support this. I'm wondering what then does support the qLaunchGDBServer command? (which it uses whenever a process is started over remote debugging).

Alternative I can start it with a cmdline to debug, but what apis do I use then to get a "process" and get started debugging?

Carlo,

You are trying to connect to a GDB server (which only debugs 1 app) with a client that is expecting a GDB remote _platform_. A platform can list, launch and attach to processes, upload/download files, it does NOT debug. There is a tool in the LLDB sources in lldb/tools/lldb-platform which implements a lldb-platform. It is not ready to be used yet, we have a platform branch where ongoing work is happening:

svn co http://llvm.org/svn/llvm-project/lldb/branches/lldb-platform-work lldb-platform

So we have done a lot of work to start getting the platforms to work as expected.

If you just want to debug, you want to use "process connect" to connect to a process that you want to debug, not a platform. See the CommandObjectProcessConnect::DoExecute() function to see how this is done. Basically if you connect to a remote debugserver and it already has a process it will be good to go. When you use "process connect", it is expected that everything you need to debug is already on the remote host.

So to sum up "platform connect" (see CommandObjectPlatformConnect) is what you tried to do.

Here are some ways you can step through the existing LLDB driver code to see things working:

mac% % /Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/debugserver localhost:3333 -- /bin/ls -lAF /tmp/
debugserver-194 for x86_64.
Listening to port 12345...

Now on your windows machine:

lldb.exe
(lldb) platform select remote-macosx
(lldb) target create --arch x86_64-apple-macosx a.out
(lldb) process connect connect://192.168.178.2:3333

Does this make sense?

Op 14-9-2012 20:35, Greg Clayton schreef:

Carlo,

You are trying to connect to a GDB server (which only debugs 1 app)
with a client that is expecting a GDB remote _platform_. A platform
can list, launch and attach to processes, upload/download files, it
does NOT debug. There is a tool in the LLDB sources in
lldb/tools/lldb-platform which implements a lldb-platform. It is not
ready to be used yet, we have a platform branch where ongoing work is
happening:

svn co
http://llvm.org/svn/llvm-project/lldb/branches/lldb-platform-work
lldb-platform

So we have done a lot of work to start getting the platforms to work
as expected.

If you just want to debug, you want to use "process connect" to
connect to a process that you want to debug, not a platform. See the
CommandObjectProcessConnect::DoExecute() function to see how this is
done. Basically if you connect to a remote debugserver and it already
has a process it will be good to go. When you use "process connect",
it is expected that everything you need to debug is already on the
remote host.

So to sum up "platform connect" (see CommandObjectPlatformConnect) is
what you tried to do.

Here are some ways you can step through the existing LLDB driver code
to see things working:

mac% %
/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/debugserver
localhost:3333 -- /bin/ls -lAF /tmp/ debugserver-194 for x86_64.
Listening to port 12345...

Now on your windows machine:

lldb.exe (lldb) platform select remote-macosx (lldb) target create
--arch x86_64-apple-macosx a.out (lldb) process connect
connect://192.168.178.2:3333

Does this make sense?

It does now, thanks... Is there something that should how to proceed from there? (from code or from lldb cmdline)? (ie what's the next step in debugging , get the process, do the first step, work with symbols?, not asking for all of them, just a starting point)

Op 14-9-2012 20:35, Greg Clayton schreef:

Carlo,

You are trying to connect to a GDB server (which only debugs 1 app)
with a client that is expecting a GDB remote _platform_. A platform
can list, launch and attach to processes, upload/download files, it
does NOT debug. There is a tool in the LLDB sources in
lldb/tools/lldb-platform which implements a lldb-platform. It is not
ready to be used yet, we have a platform branch where ongoing work is
happening:

svn co
http://llvm.org/svn/llvm-project/lldb/branches/lldb-platform-work
lldb-platform

So we have done a lot of work to start getting the platforms to work
as expected.

If you just want to debug, you want to use "process connect" to
connect to a process that you want to debug, not a platform. See the
CommandObjectProcessConnect::DoExecute() function to see how this is
done. Basically if you connect to a remote debugserver and it already
has a process it will be good to go. When you use "process connect",
it is expected that everything you need to debug is already on the
remote host.

So to sum up "platform connect" (see CommandObjectPlatformConnect) is
what you tried to do.

Here are some ways you can step through the existing LLDB driver code
to see things working:

mac% %
/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/debugserver
localhost:3333 -- /bin/ls -lAF /tmp/ debugserver-194 for x86_64.
Listening to port 12345...

Now on your windows machine:

lldb.exe (lldb) platform select remote-macosx (lldb) target create
--arch x86_64-apple-macosx a.out (lldb) process connect
connect://192.168.178.2:3333

Does this make sense?

It does now, thanks... Is there something that should how to proceed from there? (from code or from lldb cmdline)? (ie what's the next step in debugging , get the process, do the first step, work with symbols?, not asking for all of them, just a starting point)

If you are going to be building an tool that does debugging, I would suggest you compile LLDB into a dll on windwos (LLDB.dll) and work through the public LLDB API (anything that starts with "lldb::SB" (where SB stands for script bridge)). Is there a reasons you were coding with the internal API? What are you trying to do here? What is the main goal? There is a good example about how to setup an event loop to monitor a process you are debugging in:

lldb/examples/python/process_events.py

It is in python, but python also uses the exact same API as is available.

Op 14-9-2012 21:45, Greg Clayton schreef:

Thanks for all the help,

It does now, thanks... Is there something that should how to
proceed from there? (from code or from lldb cmdline)? (ie what's
the next step in debugging , get the process, do the first step,
work with symbols?, not asking for all of them, just a starting
point)

If you are going to be building an tool that does debugging, I would
suggest you compile LLDB into a dll on windwos (LLDB.dll) and work
through the public LLDB API (anything that starts with "lldb::SB"
(where SB stands for script bridge)). Is there a reasons you were
coding with the internal API?

I made some mistakes there, I'm using the internal apis now.

What are you trying to do here? What is the main goal?

Trying to make a gui (Visual Studio plugin) around LLDB for our LLVM based language.

There is a good example about how to setup an event

loop to monitor a process you are debugging in:

lldb/examples/python/process_events.py

It is in python, but python also uses the exact same API as is
available.

Cool, that worked, got it starting, continuing, and fininsh now. Only can't get it to break.
It doesn't resolve the address of _main or main, even though at this point the a.out is loaded on both the server and client, I stepped through it,

when setting the BP, inside:
size_t ObjectFileMachO::ParseSymtab (bool minimize)

"process" is null so it never seems to find any symbols.

Using this:

m_debugger = SBDebugger::Create(false);

m_target = m_debugger.CreateTarget ("\\path\\a.out", "x86_64-apple-macosx", NULL, true, m_error);

m_process = m_target.ConnectRemote(SBListener(), "connect://192.168.178.2:3333", NULL, m_error);

SBBreakpoint bp = m_target.BreakpointCreateByName("_main");

SBCommandReturnObject o;
m_commandline.HandleCommand("breakpoint list", o);
<< shows as not bound yet.

What am I missing?

Carlo Kok

Op 14-9-2012 21:45, Greg Clayton schreef:

Thanks for all the help,

It does now, thanks... Is there something that should how to
proceed from there? (from code or from lldb cmdline)? (ie what's
the next step in debugging , get the process, do the first step,
work with symbols?, not asking for all of them, just a starting
point)

If you are going to be building an tool that does debugging, I would
suggest you compile LLDB into a dll on windwos (LLDB.dll) and work
through the public LLDB API (anything that starts with "lldb::SB"
(where SB stands for script bridge)). Is there a reasons you were
coding with the internal API?

I made some mistakes there, I'm using the internal apis now.

The public API's, yes. much better.

What are you trying to do here? What is the main goal?

Trying to make a gui (Visual Studio plugin) around LLDB for our LLVM based language.

Great!

There is a good example about how to setup an event

loop to monitor a process you are debugging in:

lldb/examples/python/process_events.py

It is in python, but python also uses the exact same API as is
available.

Cool, that worked, got it starting, continuing, and fininsh now. Only can't get it to break.
It doesn't resolve the address of _main or main, even though at this point the a.out is loaded on both the server and client, I stepped through it,

Does this have anything to do with the windows paths? "\\path\\a.out"? See comments below.

when setting the BP, inside:
size_t ObjectFileMachO::ParseSymtab (bool minimize)

"process" is null so it never seems to find any symbols.

ObjectFile subclasses usually read their contents from disk, not from a process. If there is no file on disk, then LLDB will make a memory based ObjectFile that reads its data from memory, then the m_process will be valid. So the process is only needed if you are reading your object file from memory. Unless you have a complete local copy of your remote Mac system, you will end up with a lot of these, but I am guessing you have at least a local copy of your main executable "\path\a.out" right? One problem you might run into as well is all the paths in the DWARF debug info will use the "/path/a.out" style paths. I am not sure how much of this file system path stuff was modified in the windows port, but it will be essential to get this done right, so no matter what kind of path is supplied, that things "just work".

Using this:

m_debugger = SBDebugger::Create(false);

m_target = m_debugger.CreateTarget ("\\path\\a.out", "x86_64-apple-macosx", NULL, true, m_error);

Now that you have a target, you will want to verify that things actually got loaded and that LLDB was able to parse your object file "\path\a.out". Try this code just after the above line that creates the m_target:

SBModule module (m_target.FindModule(m_target.GetExecutable ()));
if (module.IsValid())
{
  SBSymbol symbol;
  const uint32_t num_symbols = module.GetNumSymbols();
  for (uint32_t i=0; i<num_symbols; ++i)
  {
    symbol = module.GetSymbolAtIndex (i);
    printf ("symbol[%u] 0x%16.16llx %s\n", i, symbol.GetName(), symbol.GetStartAddress().GetFileAddress());
  }
}

After you create your target, should _should_ have a valid symbol table for your main executable. If you don't you will want to debug info the "module.GetNumSymbols();" code (which will parse the symbol table if it already hasn't been). You might need to debug info the "m_debugger.CreateTarget()" and watch it try to create a module with your "\\path\\a.out" and make sure it succeeds.

m_process = m_target.ConnectRemote(SBListener(), "connect://192.168.178.2:3333", NULL, m_error);

SBBreakpoint bp = m_target.BreakpointCreateByName("_main");

SBCommandReturnObject o;
m_commandline.HandleCommand("breakpoint list", o);
<< shows as not bound yet.

What am I missing?

Don't add the underscore. The underscore is stripped as the symbols are added to the symbol table. If you notice, all C++ symbols start with "__Z..." (two underscores), when mangled names should only have 1 underscore. So we normalize the names for you so they are in the format you would expect. Again, it sounds like we just are not parsing your object file.

You might try having the program you run on MacOSX run to your main function and pause for 20 seconds while you then attach on windows. This will allow the process to get up and loaded and already have a bunch of shared libraries loaded. There might be issues with attaching to a process that is stopped at __dyld_start (which is what happens when you have debugserver debug an app: it starts the program stopped before it runs ANY code, so _nothing_ is loaded). If you attach after the shared libraries have loaded, we might be able to resolve symbols. Give that a try and let me know how things go.

After you attach to a running program, try doing the following code:

uint32_t num_modules = m_target.GetNumModules ();
for (uint32_t i=0; i< num_modules; ++i)
{
  module = m_target.GetModuleAtIndex (i);
  printf ("module[%u] %s %s/%s\n",
          module.GetTriple(),
          module.GetPlatformFileSpec().GetDirectory(),
          module.GetPlatformFileSpec().GetFilename());
}

This will show you what modules we discovered. If this list is empty, or just contains your original module, then we are not resolving the shared libraries, and this will make debugging not work. Again, you only expect to see if a list of actual modules if you attach to a process that has already been running, so do this with a program on the mac that ran to main and is doing a "sleep(20);", then attach with debugserver:

% ./a.out
... (a.out is sleeping in main)
% /path/to/debugserver localhost:3333 --attach <pid>

Op 17-9-2012 19:31, Greg Clayton schreef:

Cool, that worked, got it starting, continuing, and fininsh now.
Only can't get it to break. It doesn't resolve the address of _main
or main, even though at this point the a.out is loaded on both the
server and client, I stepped through it,

Does this have anything to do with the windows paths?
"\\path\\a.out"? See comments below.

the path on the "caller" does not exactly match the "server" no (can't really; windows is quite different). Any way to make that work? (I know what local path will match the server path)

when setting the BP, inside: size_t ObjectFileMachO::ParseSymtab
(bool minimize)

"process" is null so it never seems to find any symbols.

ObjectFile subclasses usually read their contents from disk, not from
a process. If there is no file on disk, then LLDB will make a memory
based ObjectFile that reads its data from memory, then the m_process
will be valid. So the process is only needed if you are reading your
object file from memory. Unless you have a complete local copy of
your remote Mac system, you will end up with a lot of these, but I am
guessing you have at least a local copy of your main executable
"\path\a.out" right? One problem you might run into as well is all
the paths in the DWARF debug info will use the "/path/a.out" style
paths. I am not sure how much of this file system path stuff was
modified in the windows port, but it will be essential to get this
done right, so no matter what kind of path is supplied, that things
"just work".

my DWARF files will have Windows paths (as that's where I compile from).

Using this:

m_debugger = SBDebugger::Create(false);

m_target = m_debugger.CreateTarget ("\\path\\a.out",
"x86_64-apple-macosx", NULL, true, m_error);

Now that you have a target, you will want to verify that things
actually got loaded and that LLDB was able to parse your object file
"\path\a.out". Try this code just after the above line that creates
the m_target:

SBModule module (m_target.FindModule(m_target.GetExecutable ())); if
(module.IsValid()) { SBSymbol symbol; const uint32_t num_symbols =
module.GetNumSymbols(); for (uint32_t i=0; i<num_symbols; ++i) {
symbol = module.GetSymbolAtIndex (i); printf ("symbol[%u] 0x%16.16llx
%s\n", i, symbol.GetName(),
symbol.GetStartAddress().GetFileAddress()); } }

After you create your target, should _should_ have a valid symbol
table for your main executable. If you don't you will want to debug
info the "module.GetNumSymbols();" code (which will parse the symbol
table if it already hasn't been). You might need to debug info the
"m_debugger.CreateTarget()" and watch it try to create a module with
your "\\path\\a.out" and make sure it succeeds.

symbol[0] 0xffffffffffffffff /Users/ck/test.m
symbol[1] 0xffffffffffffffff /var/folders/f8/pr_1fjyd1rs8p8pn_2v488j00000gn/T/te
st-NhmVzr.o
symbol[2] 0x0000000100000f30 main
symbol[3] 0x0000000100000000 _mh_execute_header
symbol[4] 0x0000000100000f56 printf
symbol[5] 0xffffffffffffffff dyld_stub_binder

Indeed, it loads.

m_process = m_target.ConnectRemote(SBListener(),
"connect://192.168.178.2:3333", NULL, m_error);

SBBreakpoint bp = m_target.BreakpointCreateByName("_main");

SBCommandReturnObject o; m_commandline.HandleCommand("breakpoint
list", o); << shows as not bound yet.

What am I missing?

Don't add the underscore. The underscore is stripped as the symbols
are added to the symbol table. If you notice, all C++ symbols start
with "__Z..." (two underscores), when mangled names should only have
1 underscore. So we normalize the names for you so they are in the
format you would expect. Again, it sounds like we just are not
parsing your object file.

Oke.

You might try having the program you run on MacOSX run to your main
function and pause for 20 seconds while you then attach on windows.
This will allow the process to get up and loaded and already have a
bunch of shared libraries loaded. There might be issues with
attaching to a process that is stopped at __dyld_start (which is what
happens when you have debugserver debug an app: it starts the program
stopped before it runs ANY code, so _nothing_ is loaded). If you
attach after the shared libraries have loaded, we might be able to
resolve symbols. Give that a try and let me know how things go.

After you attach to a running program, try doing the following code:

uint32_t num_modules = m_target.GetNumModules (); for (uint32_t i=0;
i< num_modules; ++i) { module = m_target.GetModuleAtIndex (i);
printf ("module[%u] %s %s/%s\n", module.GetTriple(),
module.GetPlatformFileSpec().GetDirectory(),
module.GetPlatformFileSpec().GetFilename()); }

This will show you what modules we discovered. If this list is empty,
or just contains your original module, then we are not resolving the
shared libraries, and this will make debugging not work. Again, you
only expect to see if a list of actual modules if you attach to a
process that has already been running, so do this with a program on
the mac that ran to main and is doing a "sleep(20);", then attach
with debugserver:

Tried the --attach thing after 20+ seconds, only gives (my windows path):

module[0] x86_64-apple-macosx C:\Users\Carlo\ConsoleApplication6\obj\Debug\a.out

(Dir = that, file = NULL)

Just one.

breakpoint list gives:
Current breakpoints:
1: name = 'main', locations = zu
   1.1: where = `main, address = (null)[0x0000000100000f30], unresolved, hit count = 0

% ./a.out ... (a.out is sleeping in main) % /path/to/debugserver
localhost:3333 --attach <pid>

It's like the server side never tells about the modules. Any place I can debug and step into to narrow this down?

Op 17-9-2012 19:31, Greg Clayton schreef:

Cool, that worked, got it starting, continuing, and fininsh now.
Only can't get it to break. It doesn't resolve the address of _main
or main, even though at this point the a.out is loaded on both the
server and client, I stepped through it,

Does this have anything to do with the windows paths?
"\\path\\a.out"? See comments below.

the path on the "caller" does not exactly match the "server" no (can't really; windows is quite different). Any way to make that work? (I know what local path will match the server path)

when setting the BP, inside: size_t ObjectFileMachO::ParseSymtab
(bool minimize)

"process" is null so it never seems to find any symbols.

ObjectFile subclasses usually read their contents from disk, not from
a process. If there is no file on disk, then LLDB will make a memory
based ObjectFile that reads its data from memory, then the m_process
will be valid. So the process is only needed if you are reading your
object file from memory. Unless you have a complete local copy of
your remote Mac system, you will end up with a lot of these, but I am
guessing you have at least a local copy of your main executable
"\path\a.out" right? One problem you might run into as well is all
the paths in the DWARF debug info will use the "/path/a.out" style
paths. I am not sure how much of this file system path stuff was
modified in the windows port, but it will be essential to get this
done right, so no matter what kind of path is supplied, that things
"just work".

my DWARF files will have Windows paths (as that's where I compile from).

Using this:

m_debugger = SBDebugger::Create(false);

m_target = m_debugger.CreateTarget ("\\path\\a.out",
"x86_64-apple-macosx", NULL, true, m_error);

Now that you have a target, you will want to verify that things
actually got loaded and that LLDB was able to parse your object file
"\path\a.out". Try this code just after the above line that creates
the m_target:

SBModule module (m_target.FindModule(m_target.GetExecutable ())); if
(module.IsValid()) { SBSymbol symbol; const uint32_t num_symbols =
module.GetNumSymbols(); for (uint32_t i=0; i<num_symbols; ++i) {
symbol = module.GetSymbolAtIndex (i); printf ("symbol[%u] 0x%16.16llx
%s\n", i, symbol.GetName(),
symbol.GetStartAddress().GetFileAddress()); } }

After you create your target, should _should_ have a valid symbol
table for your main executable. If you don't you will want to debug
info the "module.GetNumSymbols();" code (which will parse the symbol
table if it already hasn't been). You might need to debug info the
"m_debugger.CreateTarget()" and watch it try to create a module with
your "\\path\\a.out" and make sure it succeeds.

symbol[0] 0xffffffffffffffff /Users/ck/test.m
symbol[1] 0xffffffffffffffff /var/folders/f8/pr_1fjyd1rs8p8pn_2v488j00000gn/T/te
st-NhmVzr.o
symbol[2] 0x0000000100000f30 main
symbol[3] 0x0000000100000000 _mh_execute_header
symbol[4] 0x0000000100000f56 printf
symbol[5] 0xffffffffffffffff dyld_stub_binder

Indeed, it loads

Great!

m_process = m_target.ConnectRemote(SBListener(),
"connect://192.168.178.2:3333", NULL, m_error);

SBBreakpoint bp = m_target.BreakpointCreateByName("_main");

SBCommandReturnObject o; m_commandline.HandleCommand("breakpoint
list", o); << shows as not bound yet.

What am I missing?

Don't add the underscore. The underscore is stripped as the symbols
are added to the symbol table. If you notice, all C++ symbols start
with "__Z..." (two underscores), when mangled names should only have
1 underscore. So we normalize the names for you so they are in the
format you would expect. Again, it sounds like we just are not
parsing your object file.

Oke.

You might try having the program you run on MacOSX run to your main
function and pause for 20 seconds while you then attach on windows.
This will allow the process to get up and loaded and already have a
bunch of shared libraries loaded. There might be issues with
attaching to a process that is stopped at __dyld_start (which is what
happens when you have debugserver debug an app: it starts the program
stopped before it runs ANY code, so _nothing_ is loaded). If you
attach after the shared libraries have loaded, we might be able to
resolve symbols. Give that a try and let me know how things go.

After you attach to a running program, try doing the following code:

uint32_t num_modules = m_target.GetNumModules (); for (uint32_t i=0;
i< num_modules; ++i) { module = m_target.GetModuleAtIndex (i);
printf ("module[%u] %s %s/%s\n", module.GetTriple(),
module.GetPlatformFileSpec().GetDirectory(),
module.GetPlatformFileSpec().GetFilename()); }

This will show you what modules we discovered. If this list is empty,
or just contains your original module, then we are not resolving the
shared libraries, and this will make debugging not work. Again, you
only expect to see if a list of actual modules if you attach to a
process that has already been running, so do this with a program on
the mac that ran to main and is doing a "sleep(20);", then attach
with debugserver:

Tried the --attach thing after 20+ seconds, only gives (my windows path):

module[0] x86_64-apple-macosx C:\Users\Carlo\ConsoleApplication6\obj\Debug\a.out

(Dir = that, file = NULL)

Just one.

breakpoint list gives:
Current breakpoints:
1: name = 'main', locations = zu
1.1: where = `main, address = (null)[0x0000000100000f30], unresolved, hit count = 0

Yep, so we have a breakpoint we just aren't getting shared libraries to load. See below.

% ./a.out ... (a.out is sleeping in main) % /path/to/debugserver
localhost:3333 --attach <pid>

It's like the server side never tells about the modules. Any place I can debug and step into to narrow this down?

DYLD is having trouble syncing up and figuring itself out and where stuff lives in it.

You will need to debug through "DynamicLoaderMacOSXDYLD::LocateDYLD()" and let me know what path it is taking and how it is failing.

The basics of what this function will try to do:
1 - it will use a hint address given to us by "debugserver" to locate an important data structure in /usr/lib/dylb
2 - it will try and figure out the base address of dyld in memory
  I would expect this to work and I would expect us to execute and succeed with a call to DynamicLoaderMacOSXDYLD.cpp:257:
                 return ReadDYLDInfoFromMemoryAndSetNotificationCallback (m_dyld_all_image_infos.dyldImageLoadAddress);
3 - all image infos will be loaded in the above command. Debug it and let me know what is failing

Greg Clayton

Op 17-9-2012 23:23, Greg Clayton schreef:

breakpoint list gives:
Current breakpoints:
1: name = 'main', locations = zu
  1.1: where = `main, address = (null)[0x0000000100000f30], unresolved, hit count = 0

Yep, so we have a breakpoint we just aren't getting shared libraries to load. See below.

% ./a.out ... (a.out is sleeping in main) % /path/to/debugserver
localhost:3333 --attach <pid>

It's like the server side never tells about the modules. Any place I can debug and step into to narrow this down?

DYLD is having trouble syncing up and figuring itself out and where stuff lives in it.

You will need to debug through "DynamicLoaderMacOSXDYLD::LocateDYLD()" and let me know what path it is taking and how it is failing.

The basics of what this function will try to do:
1 - it will use a hint address given to us by "debugserver" to locate an important data structure in /usr/lib/dylb
2 - it will try and figure out the base address of dyld in memory
  I would expect this to work and I would expect us to execute and succeed with a call to DynamicLoaderMacOSXDYLD.cpp:257:
                  return ReadDYLDInfoFromMemoryAndSetNotificationCallback (m_dyld_all_image_infos.dyldImageLoadAddress);
3 - all image infos will be loaded in the above command. Debug it and let me know what is failing

Greg Clayton

With --attach:

shlib_addr = 0x00007fff654c1b80

if (m_process->ReadMemory (shlib_addr, buf, 4, error) == 4) << fails

Inside ReadAllImageInfosStructure the m_dyld_all_image_infos_addr ends up as 0x00007fff654c1b80 and also fails loading.

the relevant part of the socket transcript:

> $qShlibInfoAddr#00
> $7fff654c1b80#00
> $m7fff654c1a00,zx#00
< $E03#00
> $m7fff654c1a00,zx#00
< $E03#00

Yikes! Malformed memory read packets. Seem like MSVC doesn't support the the 'z' modifier for printf format strings. You will need to find all format strings that contain:

%z

The 'z' specifies the size of the same size as 'size_t', so to print a size_t value as hex we often use "%zx". On windows this is just printing "zx" to the output as we can see above in the malformed memory read packets (the "zx" should be a hex size in bytes with no hex prefix).

So find all "%zu" and "%zx" formats and replace them with "%llu" and "%llx" and cast the corresponding argument to (uint64_t).

The packets that are sent use a subclass of "Stream" called "StringStream" and we use a lot of "%zx" and "%zu" when making the GDB remote packets.

Greg Clayton

I removed all uses of "%z" modifiers in our printf format strings in top of tree. Merge top of tree over to get the fixes:

% svn commit
Sending source/API/SBBreakpoint.cpp
Sending source/API/SBCommunication.cpp
Sending source/API/SBDebugger.cpp
Sending source/API/SBFileSpec.cpp
Sending source/API/SBProcess.cpp
Sending source/API/SBValue.cpp
Sending source/Breakpoint/Breakpoint.cpp
Sending source/Breakpoint/BreakpointLocationList.cpp
Sending source/Breakpoint/WatchpointList.cpp
Sending source/Commands/CommandObjectMemory.cpp
Sending source/Core/Communication.cpp
Sending source/Core/ConnectionFileDescriptor.cpp
Sending source/Core/ConstString.cpp
Sending source/Core/EmulateInstruction.cpp
Sending source/Core/Module.cpp
Sending source/Expression/ClangFunction.cpp
Sending source/Expression/DWARFExpression.cpp
Sending source/Host/macosx/Host.mm
Sending source/Interpreter/Args.cpp
Sending source/Interpreter/OptionValueArray.cpp
Sending source/Plugins/Process/MacOSX-Kernel/CommunicationKDP.cpp
Sending source/Plugins/Process/gdb-remote/GDBRemoteCommunication.cpp
Sending source/Plugins/Process/gdb-remote/GDBRemoteCommunicationClient.cpp
Sending source/Plugins/Process/gdb-remote/ProcessGDBRemote.cpp
Sending source/Plugins/SymbolFile/DWARF/DWARFDebugAranges.cpp
Sending source/Plugins/SymbolFile/DWARF/DWARFDebugPubnames.cpp
Sending source/Plugins/SymbolFile/DWARF/SymbolFileDWARF.cpp
Sending source/Plugins/UnwindAssembly/InstEmulation/UnwindAssemblyInstEmulation.cpp
Sending source/Symbol/ClangASTContext.cpp
Sending source/Target/Memory.cpp
Sending source/Target/Process.cpp
Sending source/Target/Target.cpp
Sending source/Target/ThreadList.cpp
Sending tools/debugserver/source/DNB.cpp
Sending tools/debugserver/source/DNBBreakpoint.cpp
Sending tools/debugserver/source/MacOSX/MachException.cpp
Sending tools/debugserver/source/MacOSX/MachProcess.cpp
Sending tools/debugserver/source/MacOSX/MachTask.cpp
Sending tools/debugserver/source/MacOSX/arm/DNBArchImpl.cpp
Sending tools/debugserver/source/MacOSX/i386/DNBArchImplI386.cpp
Sending tools/debugserver/source/MacOSX/x86_64/DNBArchImplX86_64.cpp
Sending tools/debugserver/source/RNBRemote.cpp
Sending tools/debugserver/source/RNBSocket.cpp
Sending tools/debugserver/source/debugserver.cpp
Transmitting file data ............................................
Committed revision 164148.

Op 18-9-2012 20:06, Greg Clayton schreef:

I removed all uses of "%z" modifiers in our printf format strings in top of tree. Merge top of tree over to get the fixes:

Thanks. That worked (I think, it AV/sigsegvs in stacktraces now so that sounds like it has something). Merge got committed so others can use it too.

Op 18-9-2012 20:06, Greg Clayton schreef:

I removed all uses of "%z" modifiers in our printf format strings in top of tree. Merge top of tree over to get the fixes:

Thanks for the help Greg (and João Matos). Limited remote debugging works now on Windows (after my latest commit), start, continue, breakpoint, kill and event handling (Through code only, the driver is still not working on Windows).

Op 18-9-2012 21:54, Carlo Kok schreef:

Op 18-9-2012 20:06, Greg Clayton schreef:

I removed all uses of "%z" modifiers in our printf format strings in
top of tree. Merge top of tree over to get the fixes:

Thanks for the help Greg (and João Matos). Limited remote debugging
works now on Windows (after my latest commit), start, continue,
breakpoint, kill and event handling (Through code only, the driver is
still not working on Windows).

I noticed it transfers about 16mb worth of data (for a single bp on main in a printf("hello"); sleep() function). What options do I have to optimize this? It seems to be doing lots of "Read memory" from within the symbol loading code (and if I kill the process while it does it it sigsegv-s). I couldn't find a "need symbols for dylib/file X" event I could hook up to (i can get the files over and store them locally that way it could load it from disk)

Yes, all shared libraries are being read from memory.

Platforms do have the ability to set an sysroot:

(lldb) platform select --sysroot \path\to\macosx\sysroot remote-macosx

If you make a local copy of the shared libraries from your remote Mac OS (copy /usr/bin/* /usr/lib/* /System/Library/Frameworks/* /System/Library/PrivateFrameworks/* and what ever else you neeed) into a local directory like "\path\to\macosx\sysroot", then you wil avoid this issue. So when we go asking for "/usr/lib/dyld" we will look in "\path\to\macosx\sysroot\usr\lib\dyld". This is the real reason we want to have a platform connection: so we can download SDK files and cache them locally. For now we will need to just let it copy stuff.

The other way to do this, is ask users to mount the remote MacOSX root partition so that it can be accessed locally. If you enable windows sharing on your remote Mac, you should be able to connect to it, and then just give that path for the "--sysroot". We don't expose the platforms as objects through the public API at the moment, but you can easily just execute the command:

m_debugger.HandleCommand ("platform select --sysroot \path\to\macosx\sysroot remote-macosx");

m_debugger.CreateTarget(....)

Does that make sense?

Greg Clayton

Op 18-9-2012 23:25, Greg Clayton schreef:

Op 18-9-2012 21:54, Carlo Kok schreef:

Op 18-9-2012 20:06, Greg Clayton schreef:

I removed all uses of "%z" modifiers in our printf format strings in
top of tree. Merge top of tree over to get the fixes:

Thanks for the help Greg (and João Matos). Limited remote debugging
works now on Windows (after my latest commit), start, continue,
breakpoint, kill and event handling (Through code only, the driver is
still not working on Windows).

I noticed it transfers about 16mb worth of data (for a single bp on
main in a printf("hello"); sleep() function). What options do I
have to optimize this? It seems to be doing lots of "Read memory"
from within the symbol loading code (and if I kill the process
while it does it it sigsegv-s). I couldn't find a "need symbols for
dylib/file X" event I could hook up to (i can get the files over
and store them locally that way it could load it from disk)

Yes, all shared libraries are being read from memory.

Platforms do have the ability to set an sysroot:

(lldb) platform select --sysroot \path\to\macosx\sysroot
remote-macosx

If you make a local copy of the shared libraries from your remote Mac
OS (copy /usr/bin/* /usr/lib/* /System/Library/Frameworks/*
/System/Library/PrivateFrameworks/* and what ever else you neeed)
into a local directory like "\path\to\macosx\sysroot", then you wil
avoid this issue. So when we go asking for "/usr/lib/dyld" we will
look in "\path\to\macosx\sysroot\usr\lib\dyld". This is the real
reason we want to have a platform connection: so we can download SDK
files and cache them locally. For now we will need to just let it
copy stuff.

The other way to do this, is ask users to mount the remote MacOSX
root partition so that it can be accessed locally. If you enable
windows sharing on your remote Mac, you should be able to connect to
it, and then just give that path for the "--sysroot". We don't expose
the platforms as objects through the public API at the moment, but
you can easily just execute the command:

m_debugger.HandleCommand ("platform select --sysroot
\path\to\macosx\sysroot remote-macosx");

m_debugger.CreateTarget(....)

Does that make sense?

it does. First i want to try to make this work dynamically though (we have a "server app" like your "remote" host) that has the ability to download files to the computer on demand. I'm thinking I'll subclass "PlatformWindows" and override:

Error
PlatformWindows::ResolveExecutable (const FileSpec &exe_file,
                                     const ArchSpec &exe_arch,
                                     lldb::ModuleSP &exe_module_sp,
                                     const FileSpecList *module_search_paths_ptr)

That triggers for each file, and download (if possible) from there.

Forcing my users to download all files manually does not sound like a practical option for me.

Thanks for the Help,

Carlo Kok