ELF Separate Debug File question

The elf files we are debugging on Linux have split the debug information from the main binary. Something like this:

http://slackito.com/2011/08/24/separate-debug-information-with-gdb/

That leaves the main binary with a .gnu_debuglink section which points to the debug information file name, and all the debug sections in the main elf file have been stripped out.

I’m looking at implementing support for this in ObjectFileELF, and I’m thinking I can have the ObjectFileELF contain a pointer to another ObjectFileELF object when we find a .gnu_debuglink. The debug sections in that second ObjectFileELF can be added as child sections to the .gnu_debuglink section in the main binary.

I also need to get a list of paths to search for these debug files passed down to ObjectFileELF. In gdb, this is the “set debug-file-directory” command. That is all described here:

http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html

Anyone have any comments / suggestions / gotchas on this before I start heading down this path? Am I using the children section consistently with what’s expected? Any better ways to accomplish this?

Thanks much.
-Mike

Yes: Checkout the SymbolVendor class. On MacOSX we store our debug info a separate file as well which is a dSYM bundle that contains DWARF. The SymbolVendorMacOSX takes care of locating the dSYM file for a given binary. The SymbolVendor class is designed exactly for this purpose: locate and use one or more object files, other than the main object file, to give more complete debug info for a debug session. The symbol vendor may just use the Module's object file if there aren't any other files with debug info.

So in this case, you don't actually touch the ObjectFileELF plug-in at all besides possibly adding an abstract accessor to get the symbol file path(s). I believe there are probably many object file formats out there that have similar things (COFF has the same kind of thing), so we should add a "virtual FileSpecList ObjectFile::GetDebugSymbolFilePaths()" function to the main ObjectFile interface.

lldb_private::Module has two accessors: GetObjectFile() and GetSymbolVendor(). The symbol vendor currently can either use the same object file that the module uses, or it can use a different file (the extra separate ELF file in your case). If the separate debug info is Linux only, I would make a SymbolVendorLinux, or if the separate debug info is for any ELF file, then make a SymbolVendorELF. The symbol vendor then can use any tricks up its sleeves to locate the given debug info file given the actual executable object file. On MacOSX, we have a UUID in each binary that is also contained in the dSYM file. The SymbolVendorMacOSX looks right next to the binary, it uses spotlight and many other tricks to locate the debug info file using the UUID. I believe you recently added support for the 20 byte UUID values to linux, so maybe this can be used in case you move your binary to another machine at a different path so you can still be able to locate your symbol files.

So you will make one or two instances or ObjectFileELF now.

In the case where the ELF executable file contains the debug info, you make on ObjectFileELF instance when making the module, then the SymbolVendorELF will check if there is a .gnu_debuglink section, and in this case there isn't one, so it will use the same ObjectFileELF (by getting a shared pointer to the object file form the module) and use this as the object file for the debug symbols.

In the case where the ELF executable file doesn't contain the debug info, you make on ObjectFileELF instance when making the module, then the SymbolVendorELF will check if there is a .gnu_debuglink section, and in this case there _is_ one, so the SymbolVendorELF will make another instance of ObjectFileELF and use this as the object file for the debug symbols. The only tricky thing you have to do is to have this second ObjectFileELF create section/offset addresses using the sections from the Module's object file. I am not sure what these debug info files look like, or if they both have the same section definitions. On MacOSX, we have a copy of all the section definitions in the dSYM file so any symbols in the extra debug info will get mapped back to the actual sections from the main executable very easily. The dSYM file has a symbol table, the symbol table has a section index, so if and when we make symbols in this extra debug symbols object file, we use the actual section definitions from the actual object file.

So checkout the SymbolVendorMacOSX and let me know if you have any questions, but at least you have a fully working sample code to look at.

Greg Clayton

First off, thank you for your reply Greg. Very useful as usual.

The copying sections part seems less than ideal for these elf files. I've
taken a very simple app (blah) and stripped the symbols (blah.debug). The
section infos for both are down below.

blah has stripped the .debug_* sections and bumped the following sections
down. Plus the .shrstrtab (section names), .symtab (symbol table), and
.strtab are all quite different. Several sections are in blah.debug but
marked as having no data, so the offsets of the various sections would
intersect if just copied.

All those .debug sections are only used by ObjectFileELF::GetSymtab() and
the routines it calls, yes? Is there anything gained by having an entirely
separate SymbolVendorELF abstraction vs. just having ObjectFileELF do the
searching and parsing of the .debug routines always - even if that means it
loads a separate debug file? Or possibly sections could have a ObjectFileSP
so they'll know where to read their data from? I'd (obviously) really like
to avoid copying large amounts of data around...

Thanks.
-Mike

blah:
[Nr] Name Type Addr Off Size
[ 0] NULL 000000 00000 000000
[ 1] .interp PROGBITS 400238 00238 00001c
[ 2] .note.ABI-tag NOTE 400254 00254 000020
[ 3] .note.gnu.build-i NOTE 400274 00274 000024
[ 4] .gnu.hash GNU_HASH 400298 00298 004cc0
[ 5] .dynsym DYNSYM 404f58 04f58 00e550
[ 6] .dynstr STRTAB 4134a8 134a8 026a80
[ 7] .gnu.version VERSYM 439f28 39f28 00131c
[ 8] .gnu.version_r VERNEED 43b248 3b248 0000d0
[ 9] .rela.dyn RELA 43b318 3b318 0000a8
[10] .rela.plt RELA 43b3c0 3b3c0 0009a8
[11] .init PROGBITS 43bd68 3bd68 000018
[12] .plt PROGBITS 43bd80 3bd80 000680
[13] .text PROGBITS 43c400 3c400 075cf8
[14] .fini PROGBITS 4b20f8 b20f8 00000e
[15] .rodata PROGBITS 4b2120 b2120 00be00
[16] .eh_frame_hdr PROGBITS 4bdf20 bdf20 00387c
[17] .eh_frame PROGBITS 4c17a0 c17a0 01360c
[18] .gcc_except_table PROGBITS 4d4dac d4dac 004f3d
[19] .init_array INIT_ARRAY 6d9da0 d9da0 000038
[20] .ctors PROGBITS 6d9dd8 d9dd8 000010
[21] .dtors PROGBITS 6d9de8 d9de8 000010
[22] .jcr PROGBITS 6d9df8 d9df8 000008
[23] .dynamic DYNAMIC 6d9e00 d9e00 0001e0
[24] .got PROGBITS 6d9fe0 d9fe0 000008
[25] .got.plt PROGBITS 6d9fe8 d9fe8 000350
[26] .data PROGBITS 6da338 da338 0000fc
[27] .bss NOBITS 6da440 da434 000eb0
[28] .comment PROGBITS 000000 da434 00005a
[29] .gnu_debuglink PROGBITS 000000 da48e 000010
[xx]
[xx]
[xx]
[xx]
[xx]
[xx]
[30] .shstrtab STRTAB 000000 da49e 00012b
[31] .symtab SYMTAB 000000 dae10 0100b0
[32] .strtab STRTAB 000000 eaec0 02c85f

blah.debug:
[Nr] Name Type Addr Off Size
[ 0] NULL 000000 000000 000000
[ 1] .interp NOBITS 400238 000238 00001c
[ 2] .note.ABI-tag NOTE 400254 000254 000020
[ 3] .note.gnu.build-i NOTE 400274 000274 000024
[ 4] .gnu.hash NOBITS 400298 000298 004cc0
[ 5] .dynsym NOBITS 404f58 000298 00e550
[ 6] .dynstr NOBITS 4134a8 000298 026a80
[ 7] .gnu.version NOBITS 439f28 000298 00131c
[ 8] .gnu.version_r NOBITS 43b248 000298 0000d0
[ 9] .rela.dyn NOBITS 43b318 000298 0000a8
[10] .rela.plt NOBITS 43b3c0 000298 0009a8
[11] .init NOBITS 43bd68 000298 000018
[12] .plt NOBITS 43bd80 000298 000680
[13] .text NOBITS 43c400 000298 075cf8
[14] .fini NOBITS 4b20f8 000298 00000e
[15] .rodata NOBITS 4b2120 000298 00be00
[16] .eh_frame_hdr NOBITS 4bdf20 000298 00387c
[17] .eh_frame NOBITS 4c17a0 000298 01360c
[18] .gcc_except_table NOBITS 4d4dac 000298 004f3d
[19] .init_array NOBITS 6d9da0 0d9da0 000038
[20] .ctors NOBITS 6d9dd8 0d9da0 000010
[21] .dtors NOBITS 6d9de8 0d9da0 000010
[22] .jcr NOBITS 6d9df8 0d9da0 000008
[23] .dynamic NOBITS 6d9e00 0d9da0 0001e0
[24] .got NOBITS 6d9fe0 0d9da0 000008
[25] .got.plt NOBITS 6d9fe8 0d9da0 000350
[26] .data NOBITS 6da338 0d9da0 0000fc
[27] .bss NOBITS 6da440 0d9da0 000eb0
[28] .comment PROGBITS 000000 000298 00005a
[29] .debug_aranges PROGBITS 000000 0002f2 001ca0
[30] .debug_info PROGBITS 000000 001f92 295cc9
[31] .debug_abbrev PROGBITS 000000 297c5b 01409a
[32] .debug_line PROGBITS 000000 2abcf5 03881d
[33] .debug_str PROGBITS 000000 2e4512 0f32f8
[34] .debug_loc PROGBITS 000000 3d780a 1c031e
[35] .debug_ranges PROGBITS 000000 597b28 08e7d0
[36] .shstrtab STRTAB 000000 6262f8 000175
[37] .symtab SYMTAB 000000 626e30 010440
[38] .strtab STRTAB 000000 637270 02ca87

This might be a dump question but: can you just load the "blah.debug" to begin with? You would create a Module with "/path/to/blah", but when the Module asks for its object file via Module::GetObjectFile(), could it not just make an object file using "blah.debug"? This would avoid the whole issue.

The main issue would be: is there anything missing in "blah.debug" that isn't in "blah"?

This might be a dump question but: can you just load the "blah.debug" to
begin with? You would create a Module with "/path/to/blah", but when the
Module asks for its object file via Module::GetObjectFile(), could it not
just make an object file using "blah.debug"? This would avoid the whole
issue.

The main issue would be: is there anything missing in "blah.debug" that
isn't in "blah"?

Yes. blah.debug has stripped virtually all data except the debug sections.
The section headers are there, but they have the data removed and are
marked NOBITS. Plus the string sections are different.

Merging blah into blah.debug would involve copying the data, and then
merging and fixing up the string sections. Moving blah.debug into blah
would involve copying the debug sections and then merging the string
sections.

For the debug version of liblldb.so that I'm running with, the .text
section (.text data exists in liblldb.so only) is 39,875,808 bytes. The
.debug_info section (data exists in liblldb.so.debug only) is 436,187,806
bytes.

With system libraries, lldb has 26 libraries loaded when debugging my
simple helloworld sample app. Some of our games have over 130 shared
libraries loaded. I think it would be good to find a solution that doesn't
involve copying all this data around.
-Mike

Sounds like the solution needs to be that you take any NOBITS sections from "blah.debug" and find their actual counterparts in "blah". And use the sections from "blah" when resolving any actual addresses in "blah.debug". It will be kind of like the dSYM sharing of sections, except you will need to lookup the section by name in "blah" and use it in "blah.debug".

Let me know if you need more help with the SymbolVendor and the two separate ELF file approach.

Greg