LLDB does not respect platform sysroot when loading core on Linux

Eugene_Birukov1 · July 20, 2018, 6:13pm

Hello,

I would appreciate advise how to fix this correctly…

I have a core dump from somebody’s RHEL Linux and I am trying to open it on my Ubuntu. I have all the shared libraries from the target sitting under my local directory. So, GDB happily opens the core after I issue “set sysroot /path/to/local/root”, but LLDB release_60 fails to do it.

I follow instructions from Greg’s Clayton mail http://lists.llvm.org/pipermail/lldb-dev/2016-January/009236.html :

(lldb) platform select --sysroot /path/to/remote/shared/libraries remote-linux

(lldb)

Under debugger, I see that LLDB successfully created Platform object with m_sdk_root set to my path and the Target uses it as its platform:

(gdb) p target_sp->m_platform_sp->m_sdk_sysroot

$42 = {

m_string = 0x80e070 “/tmp/debugcore.3WyoW4/lib2”

}

But this value is not used when it comes to DynamicLoaderPOSIXDYLD::LoadAllCurrentModules.

(gdb) bt

#0 lldb_private::ModuleList::GetSharedModule (module_spec=…, module_sp=std::shared_ptr (empty) 0x0,

module_search_paths_ptr=0x83ad60, old_module_sp_ptr=0x7fffffffbb50, did_create_ptr=0x7fffffffbb07,

always_create=false) at /home/eugene/llvm/tools/lldb/source/Core/ModuleList.cpp:710

#1 0x00007fffedc2d130 in lldb_private::Platform::<lambda(const lldb_private::ModuleSpec&)>::operator()(const lldb_private::ModuleSpec &) const (__closure=0x8581b0, spec=…)

at /home/eugene/llvm/tools/lldb/source/Target/Platform.cpp:234

#2 0x00007fffedc34ff2 in std::_Function_handler<lldb_private::Status(const lldb_private::ModuleSpec&), lldb_private::Platform::GetSharedModule(const lldb_private::ModuleSpec&, lldb_private::Process*, lldb::ModuleSP&, const lldb_private::FileSpecList*, lldb::ModuleSP*, bool*)::<lambda(const lldb_private::ModuleSpec&)> >::_M_invoke(const std::_Any_data &, const lldb_private::ModuleSpec &) (__functor=…, __args#0=…) at /usr/include/c++/5/functional:1857

#3 0x00007fffedc37978 in std::function<lldb_private::Status (lldb_private::ModuleSpec const&)>::operator()(lldb_private::ModuleSpec const&) const (this=0x7fffffffba80, __args#0=…) at /usr/include/c++/5/functional:2267

#4 0x00007fffedc3375a in lldb_private::Platform::GetRemoteSharedModule(lldb_private::ModuleSpec const&, lldb_private::Process*, std::shared_ptr<lldb_private::Module>&, std::function<lldb_private::Status (lldb_private::ModuleSpec const&)> const&, bool*) (this=0x839330, module_spec=…, process=0x84d310, module_sp=std::shared_ptr (empty) 0x0,

module_resolver=…, did_create_ptr=0x7fffffffbb07)

at /home/eugene/llvm/tools/lldb/source/Target/Platform.cpp:1628

#5 0x00007fffedc2d2cd in lldb_private::Platform::GetSharedModule (this=0x839330, module_spec=…,

process=0x84d310, module_sp=std::shared_ptr (empty) 0x0, module_search_paths_ptr=0x83ad60,

old_module_sp_ptr=0x7fffffffbb50, did_create_ptr=0x7fffffffbb07)

at /home/eugene/llvm/tools/lldb/source/Target/Platform.cpp:240

#6 0x00007fffedc9957c in lldb_private::Target::GetSharedModule (this=0x846960, module_spec=…, error_ptr=0x0)

at /home/eugene/llvm/tools/lldb/source/Target/Target.cpp:1952

#7 0x00007fffef8e0d11 in lldb_private::DynamicLoader::LoadModuleAtAddress (this=0x9a0a70, file=…,

link_map_addr=139700267943784, base_addr=139700263510016, base_addr_is_offset=true)

at /home/eugene/llvm/tools/lldb/source/Core/DynamicLoader.cpp:171

#8 0x00007fffedd8fb55 in DynamicLoaderPOSIXDYLD::LoadAllCurrentModules (this=0x9a0a70)

at /home/eugene/llvm/tools/lldb/source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp:537

#9 0x00007fffedd8de52 in DynamicLoaderPOSIXDYLD::DidAttach (this=0x9a0a70)

at /home/eugene/llvm/tools/lldb/source/Plugins/DynamicLoader/POSIX-DYLD/DynamicLoaderPOSIXDYLD.cpp:171

#10 0x00007fffedc476d9 in lldb_private::Process::LoadCore (this=0x84d310)

at /home/eugene/llvm/tools/lldb/source/Target/Process.cpp:2853

I simply do not see any reference to sysroot in the GetSharedModule() code. What is there – it is only scanning module_search_paths_ptr looking for file. This would not work because the scan ignores the directory part of the module: it takes the next path from the list and appends the file name. What I need instead – take m_sdk_sysroot from Platform and append the full module – including directory – to it.

Unfortunately, GetSharedModule() is a static method and does not have any clue what is current platform or current target. So, should I pass another argument down there with sysroot or what? I have correct platform object at frame 4.

Thanks,

Eugene

Ted_Woodward · July 20, 2018, 7:01pm

Instead of setting the sysroot, try the command

image search-paths add / /path/to/remote/shared/libraries/

That adds to the list that the dynamic loader uses to map shared object paths.

It uses a simple text substitution, so in the above case,

/usr/lib/libc.so

Becomes

/path/to/remote/shared/libraries/usr/lib/libc.so

Matching up trailing slashes is critical, as I learned the hard way!

Ted

Eugene_Birukov1 · July 20, 2018, 7:20pm

1. Shouldn't platform trick work?
2. This doesn't work too. "image" command is rejected if I don't have the target created. But if I issue it after I got target, the module list gets practically empty. Only the modules I have exactly matching between my machine and the target are left.

Another problem: ld-linux-x86-64.so.2 has matched from my machine, but this is actually a different file.

(lldb) image search-paths add / /tmp/debugcore.3WyoW4/lib2/
error: invalid target
(lldb) target create -c core bin/executable
Core file ... (x86_64) was loaded.
(lldb) target mod list
[ 0] EB1EC4C1-8C92-1BBC-5EEC-98AAD9006691-2E1FE735 /tmp/debugcore.3WyoW4/bin/executable
[ 1] BBCFB8B3-D0D8-9B97-9231-645196845E00-CB55E7D0 /opt/xxxxx/bin/../lib/libc++.so.1
[ 2] AF1A7705-DAC0-8661-7411-8DEFE8281C64-AED273A9 /opt/xxxxx/bin/../lib/libc++abi.so.1
[ 3] 5D7B6259-5522-75A3-C17B-D4C3FD05F5A6-BF40CAA5 /lib64/ld-linux-x86-64.so.2
[ 4] F40CF56C-9984-4493-6E56-9E905018C61A-446E19F3 0x00007ffd3f9d2000 [vdso] (0x00007ffd3f9d2000)
[ 5] 402F8E9C-0000-0000-0000-000000000000 0x00007f0e8094e000 /lib64/libpthread.so.0 (0x00007f0e8094e000)
[ 6] C36E0F49-0000-0000-0000-000000000000 0x00007f0e80746000 /lib64/librt.so.1 (0x00007f0e80746000)
[ 7] DA7C854B-0000-0000-0000-000000000000 0x00007f0e80542000 /lib64/libdl.so.2 (0x00007f0e80542000)
[ 8] 9EFAADB9-0000-0000-0000-000000000000 0x00007f0e7feb2000 /lib64/libcrypto.so.10 (0x00007f0e7feb2000)
[ 9] 1B54BC7F-0000-0000-0000-000000000000 0x00007f0e7fc44000 /lib64/libssl.so.10 (0x00007f0e7fc44000)
[ 10] 0E999CDE-0000-0000-0000-000000000000 0x00007f0e7fa38000 /lib64/libnuma.so.1 (0x00007f0e7fa38000)
[ 11] 51C7A74C-0000-0000-0000-000000000000 0x00007f0e7f7ea000 /lib64/libgssapi_krb5.so.2 (0x00007f0e7f7ea000)
[ 12] D74EFC55-0000-0000-0000-000000000000 0x00007f0e7f503000 /lib64/libkrb5.so.3 (0x00007f0e7f503000)
[ 13] 57A9AB75-0000-0000-0000-000000000000 0x00007f0e7f2d1000 /lib64/libk5crypto.so.3 (0x00007f0e7f2d1000)
[ 14] 93F50DD9-0000-0000-0000-000000000000 0x00007f0e7f0cd000 /lib64/libcom_err.so.2 (0x00007f0e7f0cd000)
[ 15] 04DA7429-0000-0000-0000-000000000000 0x00007f0e7ee8f000 /lib64/libblkid.so.1 (0x00007f0e7ee8f000)
[ 16] 0D4800F4-0000-0000-0000-000000000000 0x00007f0e7ec8a000 /lib64/libsss_nss_idmap.so.0 (0x00007f0e7ec8a000)
[ 17] 07CAC506-0000-0000-0000-000000000000 0x00007f0e7ea85000 /lib64/libuuid.so.1 (0x00007f0e7ea85000)
[ 18] A69E7F20-0000-0000-0000-000000000000 0x00007f0e7e537000 /lib64/libm.so.6 (0x00007f0e7e537000)
[ 19] 9FAE7CD0-0000-0000-0000-000000000000 0x00007f0e7e321000 /lib64/libgcc_s.so.1 (0x00007f0e7e321000)
[ 20] 02E59CD0-0000-0000-0000-000000000000 0x00007f0e7df60000 /lib64/libc.so.6 (0x00007f0e7df60000)
[ 21] 9FCA7001-0000-0000-0000-000000000000 0x00007f0e7dc58000 /lib64/libstdc++.so.6 (0x00007f0e7dc58000)
[ 22] EADDCF0A-0000-0000-0000-000000000000 0x00007f0e7da42000 /lib64/libz.so.1 (0x00007f0e7da42000)
[ 23] 4158BE15-0000-0000-0000-000000000000 0x00007f0e7d833000 /lib64/libkrb5support.so.0 (0x00007f0e7d833000)
[ 24] 9213C8BE-0000-0000-0000-000000000000 0x00007f0e7d62f000 /lib64/libkeyutils.so.1 (0x00007f0e7d62f000)
[ 25] DA713559-0000-0000-0000-000000000000 0x00007f0e7d415000 /lib64/libresolv.so.2 (0x00007f0e7d415000)
[ 26] 7EC8B5FA-0000-0000-0000-000000000000 0x00007f0e7d1ee000 /lib64/libselinux.so.1 (0x00007f0e7d1ee000)
[ 27] B618A715-0000-0000-0000-000000000000 0x00007f0e7cf8d000 /lib64/libpcre.so.1 (0x00007f0e7cf8d000)
[ 28] 7BC742F6-EC7C-B8E9-3251-08C749F25E46-D76FE69D /opt/xxxxx/bin/../lib/libunwind.so.8
[ 29] A3A6D23F-0000-0000-0000-000000000000 0x00007f0e75e24000 /lib64/liblzma.so.5 (0x00007f0e75e24000)
[ 30] 682FA1FE-0000-0000-0000-000000000000 0x00007f0e2f531000 /usr/lib64/gssproxy/proxymech.so (0x00007f0e2f531000)
[ 31] DA479E71-0000-0000-0000-000000000000 0x00007f0e2f305000 /lib64/libgssrpc.so.4 (0x00007f0e2f305000)
[ 32] D10ACB70-0000-0000-0000-000000000000 0x00007f0e2b9ef000 /lib64/libnss_files.so.2 (0x00007f0e2b9ef000)
[ 33] 8A5852DC-0000-0000-0000-000000000000 0x00007f0e2b7e9000 /lib64/libnss_dns.so.2 (0x00007f0e2b7e9000)
[ 34] A19B4645-0000-0000-0000-000000000000 0x00007f0e2b733000 /lib64/libnss_myhostname.so.2 (0x00007f0e2b733000)
[ 35] CB72796B-0000-0000-0000-000000000000 0x00007f0e2b52e000 /lib64/libcap.so.2 (0x00007f0e2b52e000)
[ 36] 312FFEED-0000-0000-0000-000000000000 0x00007f0e2b2e6000 /lib64/libdw.so.1 (0x00007f0e2b2e6000)
[ 37] 1B32E461-0000-0000-0000-000000000000 0x00007f0e2b0e1000 /lib64/libattr.so.1 (0x00007f0e2b0e1000)
[ 38] 280E50A8-0000-0000-0000-000000000000 0x00007f0e2aec9000 /lib64/libelf.so.1 (0x00007f0e2aec9000)
[ 39] FFB3D20E-0000-0000-0000-000000000000 0x00007f0e2acb9000 /lib64/libbz2.so.1 (0x00007f0e2acb9000)
(lldb) image search-paths add / /tmp/debugcore.3WyoW4/lib2/
(lldb) target mod list
[ 0] EB1EC4C1-8C92-1BBC-5EEC-98AAD9006691-2E1FE735 /tmp/debugcore.3WyoW4/bin/executable
[ 1] BBCFB8B3-D0D8-9B97-9231-645196845E00-CB55E7D0 /opt/xxxxx/bin/../lib/libc++.so.1
[ 2] AF1A7705-DAC0-8661-7411-8DEFE8281C64-AED273A9 /opt/xxxxx/bin/../lib/libc++abi.so.1
[ 3] 5D7B6259-5522-75A3-C17B-D4C3FD05F5A6-BF40CAA5 /lib64/ld-linux-x86-64.so.2
(lldb)

Thanks,
Eugene

Eugene_Birukov1 · July 20, 2018, 10:55pm

I have a small fix – pass platform sysroot to ModuleList::GetSharedModule. The fix works for me.

What should I do to get it reviewed?

Thanks,

Eugene

diff --git a/include/lldb/Core/ModuleList.h b/include/lldb/Core/ModuleList.h

index 4b637c9…3214291 100644

— a/include/lldb/Core/ModuleList.h

+++ b/include/lldb/Core/ModuleList.h

@@ -541,7 +541,8 @@ public:

const FileSpecList *module_search_paths_ptr,

lldb::ModuleSP *old_module_sp_ptr,

bool *did_create_ptr,

bool always_create = false);

bool always_create = false,
const char* sysroot = nullptr);

static bool RemoveSharedModule(lldb::ModuleSP &module_sp);

diff --git a/source/Core/ModuleList.cpp b/source/Core/ModuleList.cpp

index 3970052…f2db3bb 100644

— a/source/Core/ModuleList.cpp

+++ b/source/Core/ModuleList.cpp

@@ -707,7 +707,11 @@ Status ModuleList::GetSharedModule(const ModuleSpec &module_spec,

ModuleSP &module_sp,

const FileSpecList *module_search_paths_ptr,

ModuleSP *old_module_sp_ptr,

bool *did_create_ptr, bool always_create) {

bool *did_create_ptr, bool always_create,
const char* sysroot) {
// Make sure no one else can try and get or create a module while this
// function is actively working on it by doing an extra lock on the
// global mutex list.

ModuleList &shared_module_list = GetSharedModuleList();

std::lock_guardstd::recursive_mutex guard(

shared_module_list.m_modules_mutex);

@@ -726,9 +730,6 @@ Status ModuleList::GetSharedModule(const ModuleSpec &module_spec,

const FileSpec &module_file_spec = module_spec.GetFileSpec();

const ArchSpec &arch = module_spec.GetArchitecture();

// Make sure no one else can try and get or create a module while this
// function is actively working on it by doing an extra lock on the
// global mutex list.

if (!always_create) {

ModuleList matching_module_list;

const size_t num_matching_modules =

@@ -762,7 +763,11 @@ Status ModuleList::GetSharedModule(const ModuleSpec &module_spec,

if (module_sp)

return error;

module_sp.reset(new Module(module_spec));

auto resolved_module_spec(module_spec);
if (sysroot != nullptr)
resolved_module_spec.GetFileSpec().PrependPathComponent(sysroot);

Pavel_Labath · July 23, 2018, 9:42am

Instead of "image search-paths add" you should be able to do a
"settings append target.exec-search-paths /your/sysroot". That one
should not require a running target or anything like that.

However, I agree that it would be nice for the platform sysroot to
work out of the box. Particularly, as we can then use it to do other
nice things (e.g. prevent matching of files in the main filesystem, so
we can avoid the ld-linux-x86-64.so.2 problem you encountered).

For the patch, can you create a review on the LLVM phabricator
<https://reviews.llvm.org/differential>? We can discuss the approach
there.

Eugene_Birukov1 · July 23, 2018, 5:37pm

I believe using search path is not the right approach when opening a core dump. We do not need to search for modules: we know exactly what has been loaded from where when the application crashed. The only problem is that the machine was different, so we need to prepend sysroot and get them form there. Scanning search path chain and grabbing arbitrary files that happen to have matching name would confuse the debugger.

If nobody objects, I'll prepare this change in phabricator.

clayborg · July 23, 2018, 9:57pm

If --sysroot is supplied, then it should just work as you say. The platform code will be asked to locate the file and the remote-linux platform is failing here and needs to be fixed.

Greg