LLDB problems on remote debugging

Hi LLDB devs,

I’m working on porting LLDB to work with an existing simulator(which has GDB stub, remote debugging). This simulator used to work with GDB. When using with GDB, the target file(ELF) is loaded by GDB command “load” or “remote put”.
From a LLVM talk project which is very similar to my project, their target file is loaded by the simulator itself( ./sim a.out , something like that), and lldb sets breakpoint, use “gdb-remote” command to connect to the simulator, the program starts to run immediately and stop at the breakpoint.
I can’t find any lldb command that is equal to “load” in GDB. And right now when I use “gdb-remote” to connect lldb to my simulator, lldb has command line output “Process 10115 stopped,thread #1, stop reason = signal SIGTRAP,frame #0: 0xffffffffffffffff”. Does this mean the program has already started to run? I haven’t loaded the binary.

To sum my questions:

  1. Does lldb has similar command like “load/remote put” in GDB?
  2. Does “gdb-remote” command in lldb do the “loading binary” job?
  3. Will the program start to run immediately after “gdb-remote” command in lldb?
  4. Do I have to let my simulator to load the binary by itself?

Kind regards,
Rui

Hi Rui,

What you describe is almost identical to what we do with the Hexagon simulator.

There are 2 ways to launch your program – “run” (process launch) and gdb-remote (an attach).

“run” will require some plumbing to launch your simulator the same way that lldb-server/debugserver is launched. That’s fairly involved, so I’m going to skip it for now.

For the attach case, you run your simulator, with the program on the command line. Then in a separate shell you launch lldb with the program, and run the gdb-remote command to connect to it. You’ve got this already, from your message below.

When you attach to the simulator, it comes up stopped, probably at the entrypoint, but that’s dependent on your simulator. To continue, issue a “continue” command. You’ll probably want to set a breakpoint first!

To answer your questions:

Hi LLDB devs,

I’m working on porting LLDB to work with an existing simulator(which has GDB stub, remote debugging). This simulator used to work with GDB. When using with GDB, the target file(ELF) is loaded by GDB command “load” or “remote put”.
From a LLVM talk project which is very similar to my project, their target file is loaded by the simulator itself( ./sim a.out , something like that), and lldb sets breakpoint, use “gdb-remote” command to connect to the simulator, the program starts to run immediately and stop at the breakpoint.
I can’t find any lldb command that is equal to “load” in GDB. And right now when I use “gdb-remote” to connect lldb to my simulator, lldb has command line output “Process 10115 stopped,thread #1, stop reason = signal SIGTRAP,frame #0: 0xffffffffffffffff”. Does this mean the program has already started to run? I haven’t loaded the binary.

To sum my questions:

  1. Does lldb has similar command like “load/remote put” in GDB?

Yes: “target modules load” with the --load option. Te be fair the Ted Woodward, this code didn’t exist when he was bringing up his simulator way back when, but it was later added by someone.

The “target modules load --load --file /path/to/file” can be used to load an ELF file as a dynamic loader would load the ELF file: it will only load the program headers that are PT_LOAD. You can also use the “–set-pc-to-entry” option to this command to set the PC to the entry point that is specified in the ELF file. For this to work, you ELF file has to be statically linked and all addresses should be correct in the file itself.

If you want to just blast an entire ELF file into memory and not only load PT_LOAD segments, then you can use "memory write --infile /path/to/binfile ".

Hopefully “target modules load --load” will work for you.

The code for this command is in CommandObjectTarget.cpp in CommandObjectTargetModulesLoad::DoExecute() if the second “if (load)” statement (line CommandObjectTarget.cpp :2827 for me). What it does is it will ask the ObjectFile for the file you are loading in the command (ObjectFileELF) about what its “loadables” are:

std::vectorObjectFile::LoadableData loadables(objfile->GetLoadableData(*target));

So as long as this function returns what you are interested in loading, then you should be good.

  1. Does “gdb-remote” command in lldb do the “loading binary” job?

It the invocation can do the loading, but it really depends on the GDB server. Typically there are a few ways to use GDB server:

1 - have it launch the program for you:
$ gdbserver [options] – /path/to/a.out arg1 arg2 arg2
2 - just launch a GDB server which you will connect to
$ gdbserver [options]
3 - have gdb server attach to a process for you:
$ gdbserver [options] --pid 123

When LLDB attaches to a GDB server, it will ask it if it has a process with some GDB remote packets, and if it has one, it is good to go and start debugging (solution 1 and 3 above). If it doesn’t, it can send a launch (“A” packet) or attach packet to attach to a process or wait for one (solution 2 above).

  1. Will the program start to run immediately after “gdb-remote” command in lldb?

If you have a GDB server, it will have the process stopped because the simple packet send/response doesn’t allow it to receive a packet if the process is running. So usually a GDB server will start off stopped and the debugger can auto continue or stop at this entry point (process launch --stop-at-entry 1).

  1. Do I have to let my simulator to load the binary by itself?

This is really up to your workflow and how complex your simulator is. If you are only ever loading one executable, then it might be easier to just have your simulator do this launching and stop your simulation right at the entry point and have the GDB server it vends respond to LLDB with information about the process. If you don’t have a real process ID, make sure your GDB server makes one up (like process ID of 1). If you want to see the packet traffic, then you can do this prior to sending the "gdb-remote " command in lldb:

(lldb) log enbable -f /tmp/packets.txt gdb-remote packets
(lldb) gdb-remote 1234

We have extended our GDB remote server with new packets. So be sure to read up on what your GDB server can provide, not mandatory, but many of these packets are useful and can provide detailed information to ensure LLDB selects the right targets and plug-ins. Details are in the LLDB code at:

lldb/docs/lldb-gdb-remote.txt

If you want to see how these are used, try debugging on a mac or linux and look at the packet log to see how a debug session goes.

Tricky things you might need to do:

  • make sure your GDB server details the registers that are available for your architecture. This can happen with the xmlRegisters response to the qSupported packet that will follow up with a packet that asks your GDB server for full details on your register for your emulated CPU. It can be very handy to ensure this has all of the info that is needed and allows your simulator to emulate different chips seamlessly. If your stub doesn’t support this, consider adding that support. If your stub doesn’t support his packet, then you will need to create a register definition python file and set the following setting:

(lldb) setting set plugin.process.gdb-remote.target-definition-file /path/to/_target_definition.py

Example files are available in the LLDB sources with a “_target_definition.py” suffix. For example:

lldb/examples/python/x86_64_target_definition.py

  • there are also some packets that allow your GDB server to specify where files ended up being loaded. Full blown OS plug-ins will end up using dynamic loader plug-ins that interface with the process to figure out where shared libraries have been loaded. Your GDB server can respond to the “qXfer:libraries-svr4:read” packet to tell LLDB where any files have been loaded. If you just have one file, then using “target modules load --load --file /path/to/a.out” should work and nothing more is needed.

Let us know how things go!

Greg