I’m trying to debug my cross-compiled executable on a custom linux distro created with Yocto for armv7 linked against musl. I’ll note that I can do this successfully with gdb and gdbserver but now I’m trying it with lldb.
I suspect I’m running into the exact same problems as the following issue, but there’s no mention of remote debugging so figured I should open another topic in case I’m simply doing something wrong.
The program is a singular “hello-world” main function, cross-compiled on a x86_64 host using gcc from the Yocto-generated SDK. My debugging setup is running lldb-server on the target arm platform and running lldb on the x86_64 host and initiating a remote connection. This works fine, I can connect to the server remotely.
First issue I noticed is that when following the remote debugging tutorial where here it mentions that lldb should automatically copy the target executable to the platform: Remote Debugging - 🐛 LLDB
This doesn’t seem to be happening:
# Target
lldb-server platform --server --listen "*:1234"
# Host
(lldb) log enable lldb symbol break dyld host target on-demand script source module os platform process
(lldb) platform select remote-linux
lldb force = true, arch=(<null>, <null>)
lldb create = true
Platform: remote-linux
Connected: no
(lldb) platform connect connect://169.254.17.110:1234
lldb PlatformRemoteGDBServer::GetRemoteWorkingDirectory() -> '/home/root'
Platform: remote-linux
Triple: arm-rcc-linux-musleabihf
OS Version: 5.15.163 (5.15.163-rt46)
Hostname: FAMBA1048448
Connected: yes
WorkingDir: /home/root
Kernel: #1 SMP PREEMPT Tue Apr 30 08:00:00 UTC 2024
(lldb) target list
No targets.
(lldb) file build/cross-tiny-debug-arm/a.out
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module set to triple: arm---eabihf (architecture arm)
lldb force = false, arch=(arm, arm---eabihf)
lldb create = true
lldb 0x561703c0be50 Module::Module((arm) '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out')
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module set to triple: arm---eabihf (architecture arm)
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb Target::Target created with architecture arm (arm---eabihf)
lldb 0x561703bdb5b0 Module::Module((arm) 'libstdc++.so.6')
lldb 0x561703bdb5b0 Module::~Module((unknown) '')
lldb 0x561703bdb5b0 Module::Module((arm) 'libgcc_s.so.1')
lldb 0x561703bdb5b0 Module::~Module((unknown) '')
lldb 0x561703bdb5b0 Module::Module((arm) 'libc.so')
lldb 0x561703bdb5b0 Module::~Module((unknown) '')
Current executable set to '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' (arm).
(lldb) run
lldb Target::Launch() called for /home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out
lldb Target::Launch the process instance doesn't currently exist.
lldb have platform=true, platform_sp->IsHost()=false, default_to_use_pty=false
lldb at least one of stdin/stdout/stderr was not set, evaluating default handling
lldb target stdin='(empty)', target stdout='(empty)', stderr='(empty)'
lldb Target::Launch the platform doesn't know how to debug a process, getting a process plugin to do this for us.
lldb Failed to create script object.
lldb Went to stop the private state thread, but it was already invalid.
lldb T lldb_private::ScriptedPythonInterface::Dispatch(llvm::StringRef, Status &, Args &&...) [T = std::shared_ptr<lldb_private::StructuredData::Object>, Args = <>] (is_alive) ERROR = Python object ill-formed
lldb virtual bool lldb_private::ScriptedProcessPythonInterface::IsAlive() ERROR = Null Structured Data object
lldb Went to stop the private state thread, but it was already invalid.
lldb (plugin = gdb-remote status = 6 (0x00000006), description="destroying when not connected to debugserver")
lldb Went to stop the private state thread, but it was already invalid.
lldb Went to stop the private state thread, but it was already invalid.
error: failed to launch or debug process
I don’t see any copy operations being logged, and so I suspect it’s trying to run the target locally and rightfully failing to run the cross-compiled executable.
Oddly at this point, I have been disconnected from the platform without warning:
(lldb) platform status
Platform: remote-linux
Connected: no
A few other things of note is that the target executable is loaded by lldb with architecture arm-*-*-eabihf, so right away lldb doesn’t acknowledge musl. However, the platform is correctly detected as arm-rcc-linux-musleabihf. The --arch argument for target create doesn’t accept a triple containing musleabihf. I can specify --arch arm-*-linux-eabihf but the end result is the same.
So I thought to try manually rsync-ing the executable to the arm device, starting it with lldb-server, and attaching from the host afterwards:
# Target
lldb-server platform --server --listen "*:1234" a.out
# Host
(lldb) log enable lldb symbol break dyld host target on-demand script source module os platform process
(lldb) platform select remote-linux
lldb force = true, arch=(<null>, <null>)
lldb create = true
Platform: remote-linux
Connected: no
(lldb) platform connect connect://169.254.17.110:1234
lldb PlatformRemoteGDBServer::GetRemoteWorkingDirectory() -> '/home/root'
lldb Target::SetArchitecture merging compatible arch; arch is now arm (arm-rcc-linux-musleabihf)
lldb (plugin = gdb-remote, state = stopped)
lldb (plugin = gdb-remote, state = stopped, stop_id = 1
lldb timeout = <infinite>, event_sp)...
lldb timeout = <infinite>, event_sp) => stopped
lldb Process::CompleteAttach()
lldb Target::SetArchitecture merging compatible arch; arch is now arm (arm-rcc-linux-musleabihf)
lldb Process::CompleteAttach replacing process architecture with DidAttach() architecture: arm-rcc-linux-musleabihf
lldb DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
lldb DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
lldb DYLDRendezvous::UpdateExecutablePath cannot cache exe module path: null executable module pointer
lldb DynamicLoaderPOSIXDYLD::DidAttach() pid 3690
lldb DynamicLoaderPOSIXDYLD::DidAttach pid 3690 reloaded auxv data
lldb DynamicLoaderPOSIXDYLD::ResolveExecutableModule - got executable by pid 3690: /home/root/a.out
lldb PlatformRemoteGDBServer::GetModuleSpec - failed to get module info for /home/root/a.out:arm-rcc-linux-musleabihf
lldb DynamicLoaderPOSIXDYLD::ResolveExecutableModule - failed to resolve executable with module spec "file = '/home/root/a.out', arch = arm-rcc-linux-musleabihf": '/home/root/a.out' does not exist
lldb DYLDRendezvous::UpdateExecutablePath cannot cache exe module path: null executable module pointer
lldb DynamicLoaderPOSIXDYLD::DidAttach pid 3690 executable '<null executable>', load_offset 0xffffffffffffffff
lldb after DynamicLoader::DidAttach(), target executable is (empty) (using posix-dyld plugin)
lldb Process::ShouldBroadcastEvent (0x5583186a56b0) => new state: stopped, last broadcast state: stopped - YES
lldb Process::HandlePrivateEvent (pid = 3690) broadcasting new state stopped (old state unloaded) to public
dbg.evt-handler (plugin = gdb-remote, state = stopped, restarted = 0)
lldb Process::ControlPrivateStateThread (signal = 4)
lldb Sending control event of type: 4.
intern-state Process::RunPrivateStateThread (arg = 0x558319123630, pid = 3690) thread starting...
intern-state timeout = <infinite>, event_sp)...
intern-state Process::RunPrivateStateThread (arg = 0x558319123630, pid = 3690) got a control event: 4
intern-state timeout = <infinite>, event_sp)...
Platform: remote-linux
Triple: arm-rcc-linux-musleabihf
OS Version: 5.15.163 (5.15.163-rt46)
Hostname: FAMBA1048448
Connected: yes
WorkingDir: /home/root
Kernel: #1 SMP PREEMPT Tue Apr 30 08:00:00 UTC 2024
Process 3690 stopped
* thread #1, name = 'a.out', stop reason = signal SIGSTOP
frame #0: 0x76fbe75c
-> 0x76fbe75c: bleq 0x76ffa8a0
0x76fbe760: ; unknown opcode
0x76fbe764: ldrbtmi r4, [r9], #-2307
0x76fbe768: ; unknown opcode
So this time I get some disassembly and I can step and continue, but don’t have any source, presumably because it’s not aware of the local executable. I tried adding the local file with target module add and file/target create but it didn’t help the situation.
(lldb) source list
(lldb) source info
error: The target has no associated executable images.
(lldb) target module add build/cross-tiny-debug-arm/a.out
lldb ProcessGDBRemote::GetModuleSpec - failed to get module info for build/cross-tiny-debug-arm/a.out:arm-rcc-linux-musleabihf
lldb PlatformRemoteGDBServer::GetModuleSpec - failed to get module info for build/cross-tiny-debug-arm/a.out:arm-rcc-linux-musleabihf
lldb 0x5583192ffb00 Module::Module((arm) 'build/cross-tiny-debug-arm/a.out')
lldb ObjectFileELF::GetModuleSpecifications file 'build/cross-tiny-debug-arm/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file 'build/cross-tiny-debug-arm/a.out' module set to triple: arm---eabihf (architecture arm)
lldb Found local object file but the specs didn't match
lldb 0x5583192ffb00 Module::~Module((unknown) '')
lldb ObjectFileELF::GetModuleSpecifications file 'build/cross-tiny-debug-arm/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file 'build/cross-tiny-debug-arm/a.out' module set to triple: arm---eabihf (architecture arm)
error: 'build/cross-tiny-debug-arm/a.out' does not exist
(lldb) file build/cross-tiny-debug-arm/a.out
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module set to triple: arm---eabihf (architecture arm)
lldb force = false, arch=(arm, arm---eabihf)
lldb create = true
lldb 0x5583192ffb00 Module::Module((arm) '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out')
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' module set to triple: arm---eabihf (architecture arm)
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb Target::Target created with architecture arm (arm---eabihf)
lldb 0x55831931d510 Module::Module((arm) 'libstdc++.so.6')
lldb 0x55831931d510 Module::~Module((unknown) '')
lldb 0x55831931d510 Module::Module((arm) 'libgcc_s.so.1')
lldb 0x55831931d510 Module::~Module((unknown) '')
lldb 0x55831931d510 Module::Module((arm) 'libc.so')
lldb 0x55831931d510 Module::~Module((unknown) '')
Current executable set to '/home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out' (arm).
(lldb) target list
Current targets:
target #0: <none> ( arch=arm-rcc-linux-musleabihf, platform=remote-linux, pid=3690, state=stopped )
* target #1: /home/ubuntu/dev/projects/lldb-test/build/cross-tiny-debug-arm/a.out ( arch=arm-*-*-eabihf, platform=remote-linux )
Creating the target from the local executable before running process connect didn’t make a difference, I still don’t have any source information. I’ve confirmed debug information is present in the image using image lookup -vn main a.out, and it even shows the correct local source paths.
The last thing I tried was symlinking the local executable to a local directory that matched that of target, i.e. /home/root/a.out. This is as close as I got, because it seems like lldb is correctly associating the local and remote executables, but I think it’s determining that they aren’t compatible, logging things like “Found local object file but the specs didn’t match”
(lldb) platform connect connect://169.254.17.110:1234
lldb PlatformRemoteGDBServer::GetRemoteWorkingDirectory() -> '/home/root'
lldb Target::SetArchitecture merging compatible arch; arch is now arm (arm-rcc-linux-musleabihf)
lldb (plugin = gdb-remote, state = stopped)
lldb (plugin = gdb-remote, state = stopped, stop_id = 1
lldb timeout = <infinite>, event_sp)...
lldb timeout = <infinite>, event_sp) => stopped
lldb Process::CompleteAttach()
lldb Target::SetArchitecture merging compatible arch; arch is now arm (arm-rcc-linux-musleabihf)
lldb Process::CompleteAttach replacing process architecture with DidAttach() architecture: arm-rcc-linux-musleabihf
lldb DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
lldb DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
lldb DYLDRendezvous::UpdateExecutablePath cannot cache exe module path: null executable module pointer
lldb DynamicLoaderPOSIXDYLD::DidAttach() pid 3800
lldb DynamicLoaderPOSIXDYLD::DidAttach pid 3800 reloaded auxv data
lldb DynamicLoaderPOSIXDYLD::ResolveExecutableModule - got executable by pid 3800: /home/root/a.out
lldb PlatformRemoteGDBServer::GetModuleSpec - failed to get module info for /home/root/a.out:arm-rcc-linux-musleabihf
lldb 0x562f43140480 Module::Module((arm) '/home/root/a.out')
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module set to triple: arm---eabihf (architecture arm)
lldb Found local object file but the specs didn't match
lldb 0x562f43140480 Module::~Module((unknown) '')
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module set to triple: arm---eabihf (architecture arm)
lldb 0x562f43140480 Module::Module((arm) '/home/root/a.out')
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module set to triple: arm---eabihf (architecture arm)
lldb Found local object file but the specs didn't match
lldb 0x562f43140480 Module::~Module((unknown) '')
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module OSABI: ELFOSABI_NONE
lldb ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
lldb ObjectFileELF::GetModuleSpecifications file '/home/root/a.out' module set to triple: arm---eabihf (architecture arm)
lldb DynamicLoaderPOSIXDYLD::ResolveExecutableModule - failed to resolve executable with module spec "file = '/home/root/a.out', arch = arm-rcc-linux-musleabihf, object size = 22352": '/home/root/a.out' doesn't contain any 'remote-linux' platform architectures: arm
lldb DYLDRendezvous::UpdateExecutablePath cannot cache exe module path: null executable module pointer
lldb DynamicLoaderPOSIXDYLD::DidAttach pid 3800 executable '<null executable>', load_offset 0xffffffffffffffff
lldb after DynamicLoader::DidAttach(), target executable is (empty) (using posix-dyld plugin)
lldb Process::ShouldBroadcastEvent (0x562f431976b0) => new state: stopped, last broadcast state: stopped - YES
lldb Process::HandlePrivateEvent (pid = 3800) broadcasting new state stopped (old state unloaded) to public
dbg.evt-handler (plugin = gdb-remote, state = stopped, restarted = 0)
lldb Process::ControlPrivateStateThread (signal = 4)
lldb Sending control event of type: 4.
intern-state Process::RunPrivateStateThread (arg = 0x562f43c15630, pid = 3800) thread starting...
intern-state timeout = <infinite>, event_sp)...
intern-state Process::RunPrivateStateThread (arg = 0x562f43c15630, pid = 3800) got a control event: 4
intern-state timeout = <infinite>, event_sp)...
Platform: remote-linux
Triple: arm-rcc-linux-musleabihf
OS Version: 5.15.163 (5.15.163-rt46)
Hostname: FAMBA1048448
Connected: yes
WorkingDir: /home/root
Kernel: #1 SMP PREEMPT Tue Apr 30 08:00:00 UTC 2024
Process 3800 stopped
* thread #1, name = 'a.out', stop reason = signal SIGSTOP
frame #0: 0x76fbe75c
-> 0x76fbe75c: bleq 0x76ffa8a0
0x76fbe760: ; unknown opcode
0x76fbe764: ldrbtmi r4, [r9], #-2307
0x76fbe768: ; unknown opcode
To summarize it seems like lldb cannot correctly associate the local executable with the remote one, probably because the triples do not match.
I also tried some similar processes using gdbserver on the target and (lldb) gdb-remote to connect, but I still only got as far as seeing dissasembly, and encountered some additional issues like not being to continue past a SIGTRAP in _start. I’ll spare these details and log dumps for now.
