Why is LLDB not showing debug info for my assembly file?

TL;DR Why is it that if I use clang in a two step process to assemble and link I get debug info. But if I do it in one step directly from source, I don’t?

I asked this on the discord #clang section, and on stackoverflow, but then thought that it might really be an lldb question and was directed to this platform.

The two step assemble and link looks like:

localhost % clang -g -c factorial.s       
localhost % clang -o factorial factorial.o

The one step is:

localhost % clang -g -o factorial factorial.s

For (a little) more info have a look at this SO post:

If you add -### to a clang command line, the driver will dump out the command lines it’s using to run the actual tools. Comparing the two scenarios could be informative.

Thanks for the quick response and the -### tip. Ya know, I did add -v to the command line and compared the results, but didn’t see anything obvious. I’ll take another, more thorough look though. What would you expect I’ll find?

In the two step process (the one that works) the linker uses a “-no_deduplicate” flag. This doesn’t show up as a flag in the clang 16.0 user docs. Clang command line argument reference — Clang 16.0.0git documentation

Still researching that one.

Oh, -v probably shows you the same thing. I suggest that first, to see whether the driver dropped something, mainly because it’s quick and easy not because it’s likely. Make sure you see -debug-info-kind=limited in the assembler command (the one where the first option on the line is -cc1as).

Next after that would be to poke around in the compiled binaries, looking at the object-file sections to see whether the debug info is there at all. You’re on Mac, right? so they’re MachO not ELF. Try llvm-objdump --headers <your-file-here> and see what comes up. Off the top of my head I don’t remember the MachO section names for DWARF.

Okay, I thought I had it figured out, but then not so much. BTW @pogo59 the assembler command lists -debug-info-kind=constructor not -debug-info-kind=limited. Important?

I ran clang -g -o factorial factorial.s -v to get the steps, then I copied and pasted them one by one into a terminal. The final command runs dsymutil. I did not execute that, and I had my symbols. I then executed it, got the .dSYM bundle that it generates, and lost my symbols (ironically). I then deleted that bundle, and saw that my symbols were back. Thought I had a remedy. (e.g. delete the bundle after building). So I ran clang -g -o factorial factorial.s and rm -rf factorial.dSYM and fired up lldb. No joy.

Thinking the temporary files may be an issue, I tried clang -g -o factorial factorial.s -### to get the commands without executing them. Then I executed them one by one copy/paste leaving out the last step, calling dsymutil. Success. Ran dsymutil. Broken. Ran rm -rf factorial.dSYM. Success.

BTW, I’m testing for success by launching lldb and executing source info -f factorial.s which I expect will show me something like:

[0x0000000100003f89-0x0000000100003f8b): /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s:10
[0x0000000100003f8b-0x0000000100003f8d): /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s:13
[0x0000000100003f92-0x0000000100003f93): /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s:15
[0x0000000100003f93-0x0000000100003f95): /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s:16
[0x0000000100003f95-0x0000000100003f9c): /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s:19
.
.
.

To summarize, if I execute the steps from -### myself, by hand, and delete the .dSYM bundle, it works. If I run the clang -g -o factorial factorial.s and delete the .dSYM bundle, it doesn’t work.

I must be so very close though, because this makes no sense whatsoever.

No.

It looks like this is wading into Mac-specific territory where I have no map. Tagging some folks who might know more: @jingham @JDevlieghere

What’s the output of (lldb) image list and image lookup -va $pc at a breakpoint in the file if you don’t delete the dSYM bundle?

chris@goldfish chapter11 % clang -g -o factorial factorial.s

chris@goldfish chapter11 % lldb factorial
(lldb) target create "factorial"
Current executable set to '/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial' (x86_64).
(lldb) image list
[  0] 3862A0A4-CEF3-3EFD-8CDC-EFDA916E2084 0x0000000100000000 /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial 
      /Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.dSYM/Contents/Resources/DWARF/factorial
[  1] F71FB3CA-5FCC-3577-9457-B047888A46D1 0x0000000000000000 /usr/lib/dyld 
[  2] AE5F19B4-F7F2-3993-9010-5106FB5935F3 0x00007ff80b11a000 /usr/lib/libSystem.B.dylib 
[  3] 5D79B96D-5C56-3944-A820-96A87EB3633F 0x00007ff80b115000 /usr/lib/system/libcache.dylib 
[  4] 1507C172-C6DF-3CD9-AC60-3C1283614EB9 0x00007ff80b0d1000 /usr/lib/system/libcommonCrypto.dylib 
[  5] 18CC81D0-AB47-3E33-9BBE-2C530445EB3E 0x00007ff80b0fa000 /usr/lib/system/libcompiler_rt.dylib 
[  6] 327522DC-AFF3-3153-977F-77A567533CF9 0x00007ff80b0f0000 /usr/lib/system/libcopyfile.dylib 
[  7] 884AD513-F783-39AC-9F1F-BBCECDA5C618 0x00007ff8000a9000 /usr/lib/system/libcorecrypto.dylib 
[  8] 5115C8EF-BC57-32DB-91C6-71B61A79966E 0x00007ff800167000 /usr/lib/system/libdispatch.dylib 
[  9] 4EB02A85-EC4B-3119-9CC0-FEE2B8703D0D 0x00007ff800327000 /usr/lib/system/libdyld.dylib 
[ 10] 46B050C2-0743-3ACB-A75A-3E9603F68639 0x00007ff80b10c000 /usr/lib/system/libkeymgr.dylib 
[ 11] 4EB12359-F9E1-37CF-B3EC-7353631FA3AF 0x00007ff80b0af000 /usr/lib/system/libmacho.dylib 
[ 12] 5E296613-3E5A-3C45-896E-897C93322299 0x00007ff80a730000 /usr/lib/system/libquarantine.dylib 
[ 13] 8F6D7FBF-ECDC-3BC3-B135-C6F8A5E6290B 0x00007ff80b10a000 /usr/lib/system/libremovefile.dylib 
[ 14] 6DC33D38-9D84-3237-836D-F1BCD83BA44B 0x00007ff8052bc000 /usr/lib/system/libsystem_asl.dylib 
[ 15] 13E634C6-2F3F-3BD5-B343-846E056EB792 0x00007ff800052000 /usr/lib/system/libsystem_blocks.dylib 
[ 16] E8499AA8-6800-372F-BD48-82846D88C6CB 0x00007ff8001eb000 /usr/lib/system/libsystem_c.dylib 
[ 17] 4B11756B-C75A-391D-9118-27E7A766C792 0x00007ff80b102000 /usr/lib/system/libsystem_collections.dylib 
[ 18] F2F5D775-90A8-3830-ADDA-362F37BE5767 0x00007ff8099e4000 /usr/lib/system/libsystem_configuration.dylib 
[ 19] FC8AD824-43AA-3111-BE60-76915501919A 0x00007ff808c8d000 /usr/lib/system/libsystem_containermanager.dylib 
[ 20] 626935B3-E944-379B-836D-CC6A6C569FA5 0x00007ff80ade7000 /usr/lib/system/libsystem_coreservices.dylib 
[ 21] 8B1EEBF9-8BDB-37BA-A419-4E9BD5710DA7 0x00007ff80295d000 /usr/lib/system/libsystem_darwin.dylib 
[ 22] 0569C20F-600A-3FD3-BC3B-B5924F67D4F5 0x00007ff80b10d000 /usr/lib/system/libsystem_dnssd.dylib 
[ 23] F9F60FF3-ACDD-3984-935F-9DE109C0CA02 0x00007ff8001e8000 /usr/lib/system/libsystem_featureflags.dylib 
[ 24] 8725C797-689F-3A4F-9984-AED582ACB316 0x00007ff80033d000 /usr/lib/system/libsystem_info.dylib 
[ 25] 986A748E-A553-38D6-8275-75D5CF6779F3 0x00007ff80b04d000 /usr/lib/system/libsystem_m.dylib 
[ 26] 8360F423-5EB8-317D-9CC4-3D42E94F8631 0x00007ff80013b000 /usr/lib/system/libsystem_malloc.dylib 
[ 27] 8E6A9E02-EF51-35B7-BD72-EE08DB52844F 0x00007ff805257000 /usr/lib/system/libsystem_networkextension.dylib 
[ 28] F46A1B46-55A0-3AD0-84D7-7A9C79F69C83 0x00007ff802d83000 /usr/lib/system/libsystem_notify.dylib 
[ 29] FCE09AF7-B7FB-3600-9392-E247838F217F 0x00007ff811388000 /usr/lib/system/libsystem_product_info_filter.dylib 
[ 30] FAA41CA0-1EB6-3C84-B7EB-DA4107FBBAEA 0x00007ff8099e8000 /usr/lib/system/libsystem_sandbox.dylib 
[ 31] 6F8D15B3-E715-3058-AB06-4BAC4029D4E2 0x00007ff80b107000 /usr/lib/system/libsystem_secinit.dylib 
[ 32] 792406FE-2224-3C14-BA9F-F076FD7839D2 0x00007ff8002e3000 /usr/lib/system/libsystem_kernel.dylib 
[ 33] BD2BD848-F9C9-35FF-9795-32D31977523D 0x00007ff800333000 /usr/lib/system/libsystem_platform.dylib 
[ 34] F32B6D06-B156-3DA0-B086-A31CF011362B 0x00007ff80031b000 /usr/lib/system/libsystem_pthread.dylib 
[ 35] 6EB26B7A-2FD3-3242-A1AC-606A5F45DCCE 0x00007ff806b0a000 /usr/lib/system/libsystem_symptoms.dylib 
[ 36] 3AE060F4-1265-37C4-AA7D-DA8DBD25B3CC 0x00007ff800090000 /usr/lib/system/libsystem_trace.dylib 
[ 37] 02D691BE-828E-309F-8041-F4512F5ECAE4 0x00007ff80b0dd000 /usr/lib/system/libunwind.dylib 
[ 38] 2A3239E4-3A46-3B1C-8DCF-51C34BFB200B 0x00007ff800054000 /usr/lib/system/libxpc.dylib 
[ 39] DAD86103-17E9-3016-930D-02CAD81A5ED9 0x00007ff8001ae000 /usr/lib/libobjc.A.dylib 
[ 40] 3765ECF5-76A5-3345-B52B-44F212300D81 0x00007ff8002cd000 /usr/lib/libc++abi.dylib 
[ 41] 6D7B1556-B9B2-3C3E-B2AF-F5081E41BEA0 0x00007ff80b0e8000 /usr/lib/liboah.dylib 
[ 42] 08AAA9D6-930C-3B45-A5AE-2905F436B834 0x00007ff800274000 /usr/lib/libc++.1.dylib 
(lldb) b main
Breakpoint 1: where = factorial`main, address = 0x0000000100003f89
(lldb) r
Process 35406 launched: '/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial' (x86_64)
Process 35406 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100003f89 factorial`main
factorial`main:
->  0x100003f89 <+0>: pushq  $0x0
    0x100003f8b <+2>: movq   $0x4, %rcx

factorial`pushvalues:
    0x100003f92 <+0>: pushq  %rcx
    0x100003f93 <+1>: loop   0x100003f92               ; <+0>
Target 0: (factorial) stopped.
(lldb) image lookup -va $pc
      Address: factorial[0x0000000100003f89] (factorial.__TEXT.__text + 0)
      Summary: factorial`main
       Module: file = "/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial", arch = "x86_64"
       Symbol: id = {0x00000009}, range = [0x0000000100003f89-0x0000000100003f92), name="main"

(lldb) 

and dwarfdump --name main factorial.dSYM -r 1 -p -c versus dwarfdump --name main factorial.o -r 1 -p -c?

chris@goldfish chapter11 % clang -g -o factorial factorial.s

chris@goldfish chapter11 % dwarfdump --name main factorial.dSYM -r 1 -p -c
factorial.dSYM/Contents/Resources/DWARF/factorial:      file format Mach-O 64-bit x86-64

0x0000000b: DW_TAG_compile_unit
              DW_AT_stmt_list   (0x00000000)
              DW_AT_name        ("factorial.s")
              DW_AT_comp_dir    ("/Users/chris/Dev/assembly/learning-assembly/chapter11")
              DW_AT_producer    ("Apple clang version 14.0.0 (clang-1400.0.29.102)")
              DW_AT_language    (DW_LANG_Mips_Assembler)

0x0000001e:   DW_TAG_label
                DW_AT_name      ("main")
                DW_AT_decl_file ("/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s")
                DW_AT_decl_line (11)
                DW_AT_low_pc    (0x0000000000000000)
chris@goldfish chapter11 % dwarfdump --name main factorial.o -r 1 -p -c
error: factorial.o: No such file or directory
chris@goldfish chapter11 % clang -g -c factorial.s                     
chris@goldfish chapter11 % dwarfdump --name main factorial.o -r 1 -p -c
factorial.o:    file format Mach-O 64-bit x86-64

0x0000000b: DW_TAG_compile_unit
              DW_AT_stmt_list   (0x00000000)
              DW_AT_low_pc      (0x0000000000000000)
              DW_AT_high_pc     (0x000000000000002f)
              DW_AT_name        ("factorial.s")
              DW_AT_comp_dir    ("/Users/chris/Dev/assembly/learning-assembly/chapter11")
              DW_AT_producer    ("Apple clang version 14.0.0 (clang-1400.0.29.102)")
              DW_AT_language    (DW_LANG_Mips_Assembler)

0x00000095:   DW_TAG_label
                DW_AT_name      ("main")
                DW_AT_decl_file ("/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial.s")
                DW_AT_decl_line (11)
                DW_AT_low_pc    (0x0000000000000000)
chris@goldfish chapter11 % 

It occurs to me that it is entirely possible that I’m mis-using some terminology here. By losing my symbols, I mean that when I set a break point, lldb can’t show me the source. When it is working, I expect to see my source when I break on main (for example) and the break point is hit:

chris@goldfish chapter11 % !lldb                                       
lldb factorial
(lldb) target create "factorial"
Current executable set to '/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial' (x86_64).
(lldb) b main
Breakpoint 1: where = factorial`main + 2, address = 0x0000000100003f8b
(lldb) r
Process 35823 launched: '/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial' (x86_64)
Process 35823 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100003f8b factorial`main at factorial.s:13
   10       pushq $0
   11  
   12   # push NUMBER ... 1 onto the stack
-> 13       movq $NUMBER, %rcx
   14   pushvalues:
   15       pushq %rcx
   16       loop pushvalues
Target 0: (factorial) stopped.
warning: This version of LLDB has no plugin for the language "assembler". Inspection of frame variables will be limited.
(lldb) 

What I’m seeing when “it” doesn’t work — what I’m describing as losing my symbols in a one-step build is this:

chris@goldfish chapter11 % !ll
lldb factorial
(lldb) target create "factorial"
Current executable set to '/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial' (x86_64).
(lldb) b main
Breakpoint 1: where = factorial`main, address = 0x0000000100003f89
(lldb) r
Process 35929 launched: '/Users/chris/Dev/assembly/learning-assembly/chapter11/factorial' (x86_64)
Process 35929 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
    frame #0: 0x0000000100003f89 factorial`main
factorial`main:
->  0x100003f89 <+0>: pushq  $0x0
    0x100003f8b <+2>: movq   $0x4, %rcx

factorial`pushvalues:
    0x100003f92 <+0>: pushq  %rcx
    0x100003f93 <+1>: loop   0x100003f92               ; <+0>
Target 0: (factorial) stopped.
(lldb) 

Just to be clear what’s going on here…

On Darwin systems, the linker does not ever insert DWARF debug information into a binary. Instead it uses one of two modes:

  1. Debug in .o file: leave the DWARF in the .o files and write a “debug map” into the binary that tells lldb where the .o files are and how the linker mapped symbols from .o to binary
  2. Create a dSYM (using dsymutil) which does the same linking job lldb does and produces a stand alone debug file.

The problem with the “one step” build process is that when you don’t specify a .o output, clang creates a temporary one that it deletes when the linking is done. That rules out method 1 above, so clang makes a dSYM in that case before deleting the temporary .o file. TTTT, I’ve never ensured this happens the same way with .s file but it would be surprising if it didn’t.

The dSYM should be equivalent to the .o files & debug map, if that’s not true that’s a bug in dsymutil (why Adrian was asking for info about the dSYM…)

It’s also curious that anything works on deleting the dSYM, since that implies that the .o files are still hanging around somewhere, which is unexpected.

You can view the debug map (which we cannily reused STABS entries for) by doing:

$ nm -ap <path_to_binary> | grep " OSO"

that will list all the debug files the debug map was told about. If you look at the N_FUN entries you’ll also see all the functions we were told about, etc…

You can see whether lldb read in any .o files for debugging purposes by issuing:

(lldb) image list -g

All the .o files will be at the bottom. There’s a bug in this listing (just in the listing, not the search) - we don’t list .o files we found in .a files, but I don’t think that’s relevant in your case.

Jim

Wow! I think this explains everything. Thanks for the clear and complete explanation.

It is not working exactly the same in this case. Seems there is a little daylight at the top.

That appears to be a side effect of my executing by copy/paste. They are hanging around because nobody (me) deleted them. In the “one step” build, they are sensibly removed after the build is done. But I just left them under /tmp…

Looks like the dSYM bundle info is preferred over the .o files. I used a “two step” build and then ran dsymutil, and this reverted me to the broken behavior. The factorial.o file was not in the image list. Deleting the bundle then showed my factorial.o (from the two-step build) in the image list.

Makes me wonder a little bit what’s missing from the dSYM bundle that is available in the .o file. Thanks! TIL…

The dwarfdump output fromt he .dSYM and the .o isn’t fundamentally different, so I wonder if there’s a bug in LLDB’s handling of DW_TAG_label that only exists in the dSYM SymbolFile plugin?

I looked into how to submit an issue regarding this, but it doesn’t seem to fit any of the cases outlined in How to submit an LLVM bug report.