No debug symbols in armv7-alpine-linux-musleabihf

Subprojects LLDB
1

I am currently try to use lldb and lldb-server to debug a simlple ‘hello world’ c++ program.

I started off trying to use a cross compiled toolchain but seem to be able to reproduce the issue using the system package versions of clang and lldb.

The code is compiled on and running in a container on the arm target. The arm target is running an alpine-linux 3.19 container with the clang and lldb packages installed.

I can compile and run a simple hello-world program using:

clang++ -fuse-ld=lld -g main.cpp

If I launch lldb I am able to create a target with:

target create --arch arm-*-*-eabihf --platform host a.out

giving:

* target #0: /home/fio/a.out ( arch=arm-*-*-eabihf, platform=host )

image dumps seem to give losts of information and I can set a breakpoint in main. If I run the program starts to print output to stdout. If I CTRL-C then I get assembly on the screen and if do image list:

error: the target has no associated executable images

and target list gives:

* target #0: <none> ( arch=armv7-unknown-linux-musleabihf, platform=host, pid=405, state=stopped )

2

Thanks for the report, I will attempt to reproduce it tomorrow.

3

I was able to reproduce it. clang and lldb are 17.0.5 versions installed via apk.

/ # lldb
(lldb) target create --arch arm-*-*-eabihf --platform host /tmp/test.o
Current executable set to '/tmp/test.o' (arm).
(lldb) image list
[  0] 372B6387-C6B8-B573-74C5-0D50E8A04AEC-596D3178 0x00000000 /tmp/test.o

So far so good. At least, probably, because ldd shows:

/ # ldd /tmp/test.o
        /lib/ld-musl-armhf.so.1 (0xf7f27000)
        libc.musl-armv7.so.1 => /lib/ld-musl-armhf.so.1 (0xf7f27000)

But some of that is expected I think, as lldb hasn’t seen the dynamic loader run yet.

Side note: lldb /tmp/test.o fails at run. I couldn’t figure out why, but target create seems to avoid whatever issue is causing that.

(lldb) b main
Breakpoint 1: where = test.o`main + 12 at test.c:1:14, address = 0x0000053c
(lldb) run
Process 412 launched: '/tmp/test.o' (arm)
<ctrl-c>
Process 412 stopped
* thread #1, name = 'test.o', stop reason = signal SIGSTOP
    frame #0: 0x0040053c
->  0x40053c: b      0x40053c
    0x400540: .long  0xe8bdb501                ; unknown opcode
    0x400544: ldrbmi r4, [r0, -r1]!
    0x400548: andeq  r0, r0, lr, lsr r0
(lldb) image list
error: the target has no associated executable images

Somehow process status knows the name of the file, maybe that’s just the process’ name. But no images as you saw.

What’s odd is that memory region is showing some image names.

(lldb) memory region --all
[0x0000000000000000-0x0000000000400000) ---
[0x0000000000400000-0x0000000000401000) r-x /tmp/test.o
[0x0000000000401000-0x0000000000402000) r-- /tmp/test.o
[0x0000000000402000-0x0000000000403000) rw- /tmp/test.o
[0x0000000000403000-0x0000000000404000) --- [heap]
[0x0000000000404000-0x0000000000405000) rw- [heap]
[0x0000000000405000-0x00000000f7f6b000) ---
[0x00000000f7f6b000-0x00000000f7f6c000) r-x [sigpage]
[0x00000000f7f6c000-0x00000000f7fed000) r-x /lib/ld-musl-armhf.so.1
[0x00000000f7fed000-0x00000000f7fef000) rw- /lib/ld-musl-armhf.so.1
[0x00000000f7fef000-0x00000000f7ff0000) rw-
[0x00000000f7ff0000-0x00000000fffcf000) ---
[0x00000000fffcf000-0x00000000ffff0000) rw- [stack]
[0x00000000ffff0000-0x00000000ffff1000) r-x [vectors]
[0x00000000ffff1000-0xffffffffffffffff) ---

I wonder if our hook into the dynamic loader is not working as expected. Looks like it could be.

(lldb) log enable lldb dyld
(lldb) target create --arch arm-*-*-eabihf --platform host /tmp/test.o
Current executable set to '/tmp/test.o' (arm).
(lldb) b main
Breakpoint 1: where = test.o`main + 12 at test.c:1:14, address = 0x0000053c
(lldb) run
 DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
 DynamicLoaderDarwin::UseDYLDSPI: Use old DynamicLoader plugin
 DYLDRendezvous::UpdateExecutablePath cannot cache exe module path: null executable module pointer
 DynamicLoaderPOSIXDYLD::DidLaunch()

This hook is what would tell us what the dynamic loader (ld-musl-armhf) decided to load when the program is run. Though I am surprised that a problem there would make us discard the main program image as well.

I did fix an issue with the dynamic loader on 32 bit Arm recently, where lldb would discard symbol data for the dynamic loader itself. However that didn’t cause it to throw all the other images away.

I will try building the latest lldb and confirm whether this still happens.

4

Darwin? Yeah, definitely something amiss here.

5

Part of this is that you specify arm-*-*-eabihf, which doesn’t include the OS part. That’s why we fall through to using the Darwin DYLD plugin.

arm-*-linux-eabihf fixes this problem but then fails to run the program at all. Looking into why that is the case.

6

Yes, I think I found it picked remote-linux if I didn’t specify platform as host and to do that I think arm-*-*-eabihf, was one of the only strings that worked; thanks for investigating it is much appreciated.

I was unsure if it was something to do with a mismatch in the env part of the target i.e. musleabihf vs eabihf but I don’t think I can really add much to help.

7

Yeah for whatever reason, lldb doesn’t think that the canonical triple that clang uses is compatible with the triple we derive from the object file itself.

Which is likely because lldb has 0 idea that musl specifically exists. In theory that’s not a bad thing but I bet we’re making assumptions based on glibc behaviour.

I tried the latest version and the problem is still present. A few easy hacks didn’t get me very far either. (building using instructions from APKBUILD « lldb « community - aports - Alpine packages build scripts)

It seems like we detect the file’s triple incorrectly.

/build-llvm # lldb -b -o "log enable lldb all" -o "target create /tmp/test.o" -o "run"
(lldb) log enable lldb all
(lldb) target create /tmp/test.o
 Processing command: target create /tmp/test.o
 HandleCommand, cmd_obj : 'target create'
 HandleCommand, (revised) command_string: 'target create /tmp/test.o'
 HandleCommand, wants_raw_input:'False'
 HandleCommand, command line after removing command name(s): '/tmp/test.o'
 ObjectFileELF::GetModuleSpecifications file '/tmp/test.o' module OSABI: ELFOSABI_NONE
 ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
 ObjectFileELF::GetModuleSpecifications file '/tmp/test.o' module set to triple: arm---eabihf (architecture arm)
 force = false, arch=(arm, arm---eabihf)
 create = true
 0xeb58c200 Platform::Platform()
 0xeb58c7e0 Module::Module((arm) '/tmp/test.o')
 ObjectFileELF::GetModuleSpecifications file '/tmp/test.o' module OSABI: ELFOSABI_NONE
 ObjectFileELF::RefineModuleDetailsFromNote parsing note name='GNU', type=3
 ObjectFileELF::GetModuleSpecifications file '/tmp/test.o' module set to triple: arm---eabihf (architecture arm)
 0xeb58cf90 ObjectFile::ObjectFile() module = 0xeb58c7e0 (/tmp/test.o), file = /tmp/test.o, file_offset = 0x00000000, size = 10812

That should resolve to something with Linux in it at least.

Later:

Target::Launch the platform doesn't know how to debug a process, getting a process plugin to do this for us.

On a working Ubuntu install this is followed by a message that the gdb-remote plugin will handle it. No such message here, which smells to me like we asked it if it could launch a bare metal program and it understandably, said no.

I unfortunately don’t have the time to fix this myself, so I can only suggest you try gdb which seems to work.

I will see if AArch64 Alpine has the same problem and raise a GitHub issue with a summary of what I found in case we need/want to revisit this.

Like I said, lldb has 0 specific references to musl and is not tested with it on the buildbots. That said someone is clearly building it on Alpine, and presumably running tests but maybe not on 32 bit Arm.

8

Ok, thank you for looking in to this.

We currently use gcc/gdb/gdb-server and are in the process of moving across to clang/lldb/lldb-server.

This is not a blocker for us as we can happily keep using gdb/gdb-server but would love to eventually migrate.

If you happen to hear of a workaround or fix please update this ticket and I will push on with the migration.

9

I would subscribe to the issue as well, as this thread is less likely to be updated.

Turns out it does work on arm64, but I couldn’t say why. So if anyone wants to dig into it, there’s a comparison point.

10

@DavidSpickett @alan-davies-se some things I’ve noticed working with lldb on embedded targets over the years:

  • In the case above, it looks like lldb is throwing away the target that was specified, and using the triple, etc, that it gets from lldb-server.
  • Some plugins are chatty when logging is turned on. The messages from the Darwin dyld plugin are probably being printed when lldb loops through the plugins looking for one that will handle the target. In these cases, those dyld plugins aren’t being used, but the message makes it look like they are.
  • Some older Linux tools didn’t create executables in a way that lldb can identify as Linux. Because of this, the Linux platform will match with <core>-*-* instead of just <core>-*-linux, as long as <core> is the host core.
  • When we worked on Linux for the Hexagon DSP, we used MUSL. We didn’t have any issues running programs. I even used a remote lldb on x86 Linux or Windows talking to lldb-server on Hexagon Linux to debug lldb on Hexagon Linux.