Loading debug symbols (C++, Ubuntu 14.04)

Stefan_Kratochwil · May 7, 2016, 6:45pm

Hi,

I need to gather information about variables on the stack frames of a given program, lets say apache2 (httpd), using the C++ flavour of the lldb scripting bridge.

Now, if I attach to the corresponding process id, lldb does not find any debug symbols, although I have installed the debug symbols package for apache2 (apache2-dbg).

If I use the lldb cli, I have to use
'target symbols load /usr/lib/debug/usr/sbin/apache2'
(or any other path under /usr/lib/debug, depending on the stack frame / module I am currently looking at) to load the corresponding debug symbols.

However, if I try the same with gdb, then it automagically uses the binaries under /usr/lib/debug (the non-stripped binaries from any *-dbg package are located here) for symbolification.

Now, there are two questions for me:

1.: Is there a way to let lldb automagically load debug symbols from /usr/lib/debug, if there are any?

2.: If not, how can I do this manually using the C++ API?
The SBTarget class does not have suitable methods for this - the closest thing is SBTarget#AddModule(), but this gives me a _new_ module, instead of associating a symbolfile with an existing one. The code path of CommandObjectTarget.cpp#AddModuleSymbols() seems not to be reflected in the scripting bridge, or at least I haven't found it yet.

I am not using the most recent lldb version - I am currently using svn rev 247535, my operating system is a Ubuntu 14.04.

Any help on this is very appreciated!

Thank you very much, and best regards,
Stefan Kratochwil

Pavel_Labath · May 9, 2016, 8:22am

Hi Stefan,

what you are describing should work out of the box, so the fact that
you are having to add the symbols manually is a bug. I haven't tried
it with the apache binary specifically, but lldb can certainly find
external debug symbols for libc (definitely on ubuntu 14.04, as that's
what I use). We need to figure out what is different in your case. The
most important function here is SymbolVendorELF::CreateInstance. Could
you step through that function, and see why it fails to find the file?

Stefan_Kratochwil · May 9, 2016, 9:00am

Hi Pavel,

thanks for your quick reply. I am currently using a release build, so it may take a moment until I can step through there. I'll report back as soon as possible (my masters thesis somehow depends on that ).

I was just about to implement an addon for the API tonight, when I found SBDebugger::HandleCommand("target symbols add ..."). Yeah, it's a little weird way of using an API, and it's a one way command with stdout as reply channel, but it works ^^

Cheers,
Stefan

Pavel_Labath · May 9, 2016, 9:23am

You can also use SBCommandInterpreter::HandleCommand, which gives you
a bit more control over what happens.

good luck with your thesis.

pl

Stefan_Kratochwil · May 9, 2016, 12:29pm

Hi Pavel,

I stepped through SymbolVendorELF::CreateInstance(), and it seems that the problem lies within Symbols::LocateExecutableSymbolFile (Host/common/Symbols.cpp). This method looks at the correct locations:

(gdb) print debug_file_search_paths
$35 = {m_files = std::vector of length 3, capacity 4 = {
   {m_directory = {m_string = 0x7994f8 "/usr/lib"},
    m_filename = {m_string = 0x7fffe400ba88 "x86_64-linux-gnu"},
    m_is_resolved = true,
    m_syntax = lldb_private::FileSpec::ePathSyntaxPosix},

   {m_directory = {m_string = 0x7fffe400bae0 "/home/stefan/git/dsu-ld/libdsu-project"},
    m_filename = {m_string = 0x7fffe400bac8 "."},
    m_is_resolved = true,
    m_syntax = lldb_private::FileSpec::ePathSyntaxPosix},

   {m_directory = {m_string = 0x7994f8 "/usr/lib"},
    m_filename = {m_string = 0x7fffe400bb18 "debug"},
    m_is_resolved = true,
    m_syntax = lldb_private::FileSpec::ePathSyntaxPosix}}}

But it looks for the wrong file names. Under Ubuntu, the contents of a *-dbg packge are stored under /usr/lib/debug, using the exact same file names as their regular executable counterparts. In the case of apache2-dbg, the files under /usr/lib/debug are (for example):

/usr/lib/debug/usr/bin/rotatelogs
/usr/lib/debug/usr/sbin/apache2
/usr/lib/debug/usr/lib/apache2/modules/mod_ssl.so

and so on.

If i now look at the file names lldb looks for, I find for /usr/lib/x86_64-linux-gnu:
(gdb) print files
$36 = std::vector of length 4, capacity 4 = {
"/usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
"/usr/lib/x86_64-linux-gnu/.debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
"/usr/lib/x86_64-linux-gnu/.build-id/B5/9D72EE0B01E9B2947C7F95E7B4EBBBF05A8AA5.debug",
"/usr/lib/x86_64-linux-gnu/usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug"}

For my /home/stefan/git/dsu-ld/libdsu-project/:
(gdb) print files
$38 = std::vector of length 4, capacity 4 = {
"/home/stefan/git/dsu-ld/libdsu-project/./9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
"/home/stefan/git/dsu-ld/libdsu-project/./.debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
"/home/stefan/git/dsu-ld/libdsu-project/./.build-id/B5/9D72EE0B01E9B2947C7F95E7B4EBBBF05A8AA5.debug",

"/home/stefan/git/dsu-ld/libdsu-project/./usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug"}

And finally for the /usr/lib/debug directory:
(gdb) print files
$39 = std::vector of length 4, capacity 4 = {
"/usr/lib/debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
"/usr/lib/debug/.debug/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug",
"/usr/lib/debug/.build-id/B5/9D72EE0B01E9B2947C7F95E7B4EBBBF05A8AA5.debug",
"/usr/lib/debug/usr/lib/x86_64-linux-gnu/9d72ee0b01e9b2947c7f95e7b4ebbbf05a8aa5.debug"}

So, lldb expects that the uuid/build of the Module it is about to load is also contained in the symbol file name.

Imho, we also have to take the FileSpec contents of the ModuleSpec we are currently looking at, and add its path to the files vector. But I am not that familiar with the whole lldb backend to tell for sure what would happen then.

I also checked the current HEAD of the lldb source tree to see if there are any changes, compared with my older revision. There is only a slight difference in how Symbols::LoadExecutableSymbolFile uses the /usr/lib/debug directory:

// Some debug files may stored in the module directory like this:
// /usr/lib/debug/usr/lib/library.so.debug
if (!file_dir.IsEmpty())
files.push_back (dirname + file_dir.AsCString() + "/" ++ symbol_filename);

This would be in fact the solution to my problem, if it wouldn't use the symbol_filename (which contains a uuid, something the *-dbg packages don't have).

Well, what do you think?

Cheers,
Stefan

Greg_Clayton1 · May 9, 2016, 5:47pm

I also checked the current HEAD of the lldb source tree to see if there are any changes, compared with my older revision. There is only a slight difference in how Symbols::LoadExecutableSymbolFile uses the /usr/lib/debug directory:

// Some debug files may stored in the module directory like this:
// /usr/lib/debug/usr/lib/library.so.debug
if (!file_dir.IsEmpty())
files.push_back (dirname + file_dir.AsCString() + "/" ++ symbol_filename);

This would be in fact the solution to my problem, if it wouldn't use the symbol_filename (which contains a uuid, something the *-dbg packages don't have).

Well, what do you think?

That sounds like a valid plan.

Pavel_Labath · May 11, 2016, 11:30am

Hi,

thank you for investigating this.

So, what you are describing should normally work. Generally
obj_file->GetDebugSymbolFilePaths() (SymbolVendorELF.cpp:99) should
return the correct name to look for the debug file. E.g., in my case
for libpthread it returns "libpthread-2.19.so", and indeed the debug
file is found at
"/usr/lib/debug/lib/x86_64-linux-gnu/libpthread-2.19.so". This
information should come from the .gnu_debuglink section on the main
file. Could you check what does your file specify (e.g., something
like readelf -p .gnu_debuglink /lib/x86_64-linux-gnu/libpthread.so.0).
If the correct file name is listed there, then we need to figure out
why it's not picked up. If it has an incorrect file, then we need to
find another way to locate the file)

pl