For binaries build with split dwarf, is there any best practice for distributing dwps?
[DWARF][DWP] 4GB limit mentioned how google deals with collecting profiles that rely on debug information for binaries build with split dwarf.
IIUC, at google when deploying binary in production dwp packages don’t come along with the binary.
In my case, we do the same thing, dwp don’t go with binary to the production systems. But when debugging a running process or core dumps with gdb/lldb, users need to download dwp by themselves which is not convenient.
I know there is a debuginfd support in gdb. If gdb is built with debuginfod, gdb can automatically download DWARF from servers.
So if I extend debuginfod to support dwp, gdb can automatically download dwp from servers when debugging binaries build with split dwarf. I’m not sure if this a right way.
Or maybe just distributing binary with dwp together is more simple way?
Looking for suggestions and best practice. Thanks in advance.
@petrhosek - could you answer a question on debuginfod, or have some folks who own that in your area these days?
Does debuginfod already support dwp? Or perhaps debuginfod is agnostic to dwp files (like it’s build-id + arbitrary filename?)?
Oh, given the existence of lldb/test/API/debuginfod/SplitDWARF/TestDebuginfodDWP.py - I guess at least lldb has support for DWP files with debuginfod, so that’s a good sign.
To the original question: debuginfod seems like a fine way to lookup a dwp file for an executable.
What’s easiest I guess depends on your distribution story/needs - if your users or costs are filesize sensitive for whatever reason, you probably want to distribute stripped binaries to them anyway - so they’ll have /some/ extra step to get the debug info, either just an unstripped binary or an unstripped binary and dwp.
Someone internally worked on the debuginfod server, from my understanding it works with DWP file.
For stripping the binary, using objcopy. Some care needs to be taken. Because result will be <binary> .debuginfo.<binary> <dwp file>
Latter needs to be named carefully since it is found by name. I think it will need to be .debuginfo.<binary>.dwp. This also might affect symbolization if the tool doesn’t know how to deal/find .debuginfo it won’t find line number information.
I agree that debuginfod is the ideal mechanism for distributing .dwp. debuginfod supports three primary endpoints:
/buildid/BUILDID/debuginfo returns a binary object that contains the customary .*debug_* sections. This may be a split debuginfo file as created by strip, or it may be an original unstripped executable.
/buildid/BUILDID/executable returns a binary object that contains the normal executable segments. This may be a executable stripped by strip, or it may be an original unstripped executable.
/buildid/BUILDID/source/SOURCE/FILE returns a a binary object that contains the source file mentioned.
To support split DWARF, one possibility might be for /buildid/BUILDID/debuginfo to return the .dwp file and for /buildid/BUILDID/executable to return the unstripped file.
That approach wouldn’t handle the case where the binary is stripped to further reduce the size by omitting the skeleton sections.
To support such use case, we might consider introducing a new endpoint /buildid/BUILDID/dwp to return the .dwp file. That would require discussion with the elfutils maintainers.
I think we should also consider support for .dwp with the “build ID” lookup method. The convention there is to use /path/to/.build-id/ab/cdef1234.debug for the unstripped file corresponding to /path/to/.build-id/ab/cdef1234 which is the stripped file. Same as for debuginfod, we may want to additionaly support /path/to/.build-id/ab/cdef1234.dwp for the .dwp file.
Most of the tools that use debuginfod have two reasons to perform a debuginfod lookup:
--build-id is presented on the command line in lieu of a binary to operate on.
The tool needs or could benefit from debug sections, but none were found in the binary passed on the command line.
In the both cases the tool will do a lookup using the debuginfo endpoint. In the first case the result is used as the executable to operate on, and in the latter case the result is used just to obtain debug sections.
Providing .dwp as /debuginfo should work fine for case 2, but it might cause problems in case 1 if the tool needs access to the executable sections. Arguably this is an issue with the implementation; it should probably also fetch /executable if it detects that no executable sections are present in what it fetched from /debuginfo.
Sorry for the late response. I have buffer to back on this now…
Oh, given the existence of lldb/test/API/debuginfod/SplitDWARF/TestDebuginfodDWP.py - I guess at least lldb has support for DWP files with debuginfod, so that’s a good sign.
Thanks for pointing out lldb/test/API/debuginfod/SplitDWARF/TestDebuginfodDWP.py - From the test, it seems that lldb has support for debuginfod providing DWP files as /debuginfo.
What’s easiest I guess depends on your distribution story/needs - if your users or costs are filesize sensitive for whatever reason, you probably want to distribute stripped binaries to them anyway - so they’ll have /some/ extra step to get the debug info, either just an unstripped binary or an unstripped binary and dwp.
In my case, we are not filesize sensitive, so I think just unstripped binary and dwp is a reasonable approach.
If you don’t mind me asking, are you distributing stripped binary by default? And unstripped binary/separate debuginfo file and DWP file are downloaded when needed, e.g. when debugging with gdb/lldb, symbolization for PGO?