Split Symbol lookup on remote target

Subprojects LLDB
1

Hello,

there are 2 options that seem relevant:
target.debug-file-search-paths and symbol.debug-info-symlink-paths

Unforunately I dont know how to correctly use either, as I am unable to find docs describing the lookup.
Is it supposed to be a prefix, supposed to be a structure based on build-uuid or something completely different?

The libraries on the filesystem look like this:

/usr/lib/libc.so.6 → libc-2.28.so
/usr/lib/libc-2.28.so .gnu_debuglink=libc-2.28.so.debug
/usr/local/lib/debug/usr/lib/libc-2.28.so.debug

ldd would resolve the dependency with /lib64/libc.so.6

I was not able to get lldb to resolve the split debug file.

Among the things I tried:

settings set symbols.debug-info-symlink-paths [/tmp/copied_to_host/usr/local/lib/debug]
settings set target.debug-file-search-paths [/usr/local/lib/debug, /usr/local/lib/debug/usr/lib]

disabling platform.use-module-cache will further remove everything from image list

2

I’ve been working with this option lately https://reviews.llvm.org/D157609. It is a set of prefix paths that we (after that patch) will append various things to in an attempt to find the DWO files.

Despite the name, we do not currently use that setting to find .dwo files.

For DWP files we do a better job but I’m not 100% sure what paths we try.

Can you find the .dwo/.dwp files on the remote yourself, just so we know what answer we want lldb to get? It may not be possible with lldb as it stands I’m afraid.

3

I am not using .dwo files, what I meant was that I split of the elf sections with debug-info like this:

install -p -D -m644 /usr/lib/libc-2.28.so /usr/local/lib/debug/usr/lib/libc-2.28.so.debug
$OBJCOPY -R .gnu_debuglink --add-gnu-debuglink /usr/local/lib/debug/usr/lib/libc-2.28.so.debug  /usr/lib/libc-2.28.so
$STRIP --remove-section=.comment --remove-section=.note --strip-unneeded /usr/lib/libc-2.28.so

They arent needed on production systems and can be latter added back or thew debugger ideally could grab them from the host system, while debugging the target remotely.

With gdb I can do this with set debug-file-directory target:/usr/local/lib/debug (or a direct path on my host system)

Can you find the .dwo/.dwp files on the remote yourself, just so we know what answer we want lldb to get?

I cant answer this, seems to be part of our misunderstanding

4

Right, I am incorrectly thinking of split DWARF as in the feature in DWARF. This is the use of gnu-debuglink instead.

Unfortunately I’m not familiar with this technique but looking at look at our tests, we do have some that test its use. So I assume we support it.

The likely codepath is SymbolVendorELF::CreateInstance which passes target.debug-file-search-paths to LocateExecutableSymbolFile.

One thing that was pointed out on my patch was that if this setting is set after loading the program file, it may not have the intended effect. So you could trying setting it earlier if possible.

Otherwise I recommend reading through that code path and trying to make a local reproducer to see if this is something caused by the remote connection.

5

That’s way to much to dig through. From a glance it seems that the behaviour is only implemented for Mac.

6

Another possibility is to load libc.so.6 manually with target modules add.

First, figure out what the dynamic loader says the file is. Run your program, stop it, and type “target modules list”. That will tell you which filename the loader tells lldb it is using. Rename your copy with symbols to that name, then quit lldb, run it again with your program, and “target modules add ”. Then run the program and see if lldb can resolve the remote version with the local version. If it does, it will slide the symbols for you. If it doesn’t, you’ll need to use “target modules load -s ” to slide the symbols to the shared library’s load address.

7

@tedwoodward : Thats a workaround, not something I would want have to do manually.

I would like to know how lldb is supposed to lookup debug libraries.

  • fetch build-ids from the loaded libs, then use debuginfod for lookup
  • search in prefix-dirs (like gdb does with debug-file-directory)
  • search in build-id dirs (ex /usr/local/lib/debug/.build-id/0b/1a0cce8fe198296b4ddb773b66f06cca7d7f3a.debug)

Theres no doc and no way to trace the lookup?

8

You can try the logs (lldb) log enable lldb all and then narrow it from there. At a glance there isn’t much logging in the search functions though, it’s something we could do with improving.

We do not have documentation of the process either beyond scattered parts of settings help for the various options. The Mac stuff probably has more info than anything else.

9

There are different issues compounding this:

  • The shared library is accessed using symlinks, so (even though ldd says /lib64/libc.so.6) we’re not 100% sure what the loader will populate the link map with
  • You’re debugging a remote system, so lldb can’t find the correct library on the host system

First, you need to know what file the loader puts in the link map. The right log for this is the dynamic loader plugin log (‘log enable lldb dyld’), but the plugin that’s loaded for your system might not have verbose enough logging to show this. You can try that, or do the steps I suggested above, to find out the filename that the loader is publishing.

Once you know the filename that lldb is told is being loaded, you need to point lldb to it. The command to do this is “target modules search-paths add”, which you can use to tell lldb to load libraries that are published at a given path from a different path.

For example, if your program is loading /remote/path/lib/libfoo.so and you’ve got libfoo.so on the host system at /host/path/build/lib/libfoo.so, you would say “target modules search-paths add /remote/path /host/path/build”. This is a textual substitution.