process calling lldb to symbolicate its own backtrace

Timothee_Cour2 · September 24, 2013, 7:32am

I’m trying to get complete stacktraces from C/C++ (backtrace/backtrace_symbols has issues, eg line numbers are often wrong and file name omits full path), and want to use lldb for that.

from the main process A (with pidA), I tried calling lldb in a separate process that attaches to pidA and then prints the backtrace; that works but is slow.

So I’d like A to directly call lldb (via the C++ api) without having to launch a separate process, but can’t make it work so far:

SBDebugger.Create (works)

CreateTargetWithFileAndArch (works)

FindFunctions (works but then GetContextAtIndex(0).GetLineEntry is invalid)
I also tried using ResolveFileAddress but that didn’t work; not sure if it’s because I need to set the load address somehow.

Note that all the above work when calling a separate process.

Is there some example code snippet for doing that?

Greg_Clayton1 · September 24, 2013, 5:10pm

In order to symbolicate here are the steps:

1 - Create debugger
2 - Create target with the following call:

lldb::SBTarget
SBDebugger::CreateTarget (const char *filename,
                          const char *target_triple,
                          const char *platform_name,
                          bool add_dependent_modules,
                          lldb::SBError& sb_error)

You can specify NULL for platform if it is the local host platform, else specify a valid platform name ("remote-linux", "remote-macosx", etc).

Specify "false" for "add_dependent_modules. This will make a target that _only_ loads the files you add to it.

3 - Load an additional modules (shared libraries) you need to do your symbolication with:

lldb::SBModule
SBTarget::AddModule (const char *path,
                     const char *triple,
                     const char *uuid_cstr,
                     const char *symfile)

Lets say you have "/tmp/a.so" which doesn't contain debug symbols, but "/tmp/debug/a.so" does and is a separate stand alone debug info file, you would specify "/tmp/a.so" for "path" and "/tmp/debug/a.so" for "symfile". Specify NULL for "uuid_cstr" unless you have a crash log that has the UUID information for each module in it. Use the same "triple" as you did for the target create.

4 - Specify the load locations for the sections you care about for each module

First find the module:

SBFileSpec module_path("/tmp/a.so");
SBModule module = target.FindModule(module_path);
if (module.IsValid())
{

Now you need to decide if you want to rigidly slide your executables, or set the load address of sections individually. If you know the slide of your library and all sections should be slid by a fixed amount, figure out the slide and slide it:

int64_t slide = ...
target.SetModuleLoadAddress (module, slide);

Or, you can lookup the sections and set the load addresses individually:

    lldb::addr_t text_load_address = ...;
    SBSection section = module.FindSection(".text")
    target.SetSectionLoadAddress(section, text_load_address);

When setting the load address of sections, you don't need to compute a slide, you just need to set the actual address itself for the base of the section.

Repeat this for all modules so they all have load locations and now you are ready to do lookups:

lldb::addr_t lookup_addr = ...;

    lldb::SBAddress addr = target.ResolveLoadAddress (lookup_addr);
    if (addr.GetSection())
    {
        // If we have a valid section, then we can symbolicate
        SBSymbolContext sc = target.ResolveSymbolContextForAddress (addr, lldb::eSymbolContextEverything);

Now you have a symbol context "sc" which has everything filled in. The symbol context has accessors for everything you need:

    lldb::SBModule GetModule ();
    lldb::SBCompileUnit GetCompileUnit ();
    lldb::SBFunction GetFunction ();
    lldb::SBBlock GetBlock ();
    lldb::SBLineEntry GetLineEntry ();
    lldb::SBSymbol GetSymbol ();

Each of these objects could be valid, or could be invalid. Test each one with "IsValid()":

    if (GetFunction().IsValid())
    {
    }
    else if (GetSymbol().IsValid())
    {
    }

Now you have the deepest most scope for the address you have looked up. You might want to figure out if there are any inlined functions. For this you can use:

    SBSymbolContext
    SBSymbolContext::GetParentOfInlinedScope (const SBAddress &curr_frame_pc,
                                              SBAddress &parent_frame_addr) const;

For example, we already have "addr" from above which is a SBAddress:

SBAddress curr_addr(addr);

    while (1)
    {
        SBAddress parent_addr;
        SBSymbolContext parent_sc = sc.GetParentOfInlinedScope(curr_addr, parent_addr);
        if (!parent_sc.IsValid())
            break;
        // Print out frame info in "parent_addr" and "parent_sc"

}

There are more basics on our website:

http://lldb.llvm.org/symbolication.html

Timothee_Cour2 · September 24, 2013, 7:03pm

Thanks for your answer, I understand all the steps except for how to determine slide address or load address. The docs in http://lldb.llvm.org/symbolication.html weren’t very helpful for determining that (eg --slide 0x123000 : not sure where to determine that).

Greg_Clayton1 · September 24, 2013, 8:05pm

Thanks for your answer, I understand all the steps except for how to determine slide address or load address. The docs in http://lldb.llvm.org/symbolication.html weren't very helpful for determining that (eg --slide 0x123000 : not sure where to determine that).

In order to make sense of your backtraces, you will need to have access to the shared library load locations. The symbolication steps I outlined were for having a backtrace from a binary that is no longer running and you are later trying to symbolicate things. In order to do this, you need to have access to the load locations of the binaries (MacOSX crash logs have all of this information).

You don't need to do any of the steps 1 - 4 if you are attaching to a process. If you can attach to the process, get the target and this target will automatically have a complete list of loaded shared libraries already slid around and ready to start doing lookups (resolve the SBAddress from a load address, then resolve the symbol context).

So it really depends on where you are getting the addresses that you want to symbolicate. If you get a backtrace, wait for a while, then attach to a process so you can symbolicate, the process might have loaded/unloaded one or more shared libraries in the time gap since the backtrace was taken and there are no guarantees that the addresses will resolve to the right symbols.

If you are trying to use LLDB within your process, you won't be able to attach to yourself. We also don't have a mode where LLDB attaches for viewing only (not for debugging). This kind of viewing only attach would then allow LLDB to attach to the process in which it is running. If you want to use LLDB for this kind of scenario, you will need to save the stack backtraces and also find a way to save all of the shared library load locations so you can later symbolicate with a separate process.

Timothee_Cour2 · September 24, 2013, 8:47pm

> Thanks for your answer, I understand all the steps except for how to
determine slide address or load address. The docs in
http://lldb.llvm.org/symbolication.html weren't very helpful for
determining that (eg --slide 0x123000 : not sure where to determine that).

In order to make sense of your backtraces, you will need to have access to
the shared library load locations. The symbolication steps I outlined were
for having a backtrace from a binary that is no longer running and you are
later trying to symbolicate things. In order to do this, you need to have
access to the load locations of the binaries (MacOSX crash logs have all of
this information).

You don't need to do any of the steps 1 - 4 if you are attaching to a
process. If you can attach to the process, get the target and this target
will automatically have a complete list of loaded shared libraries already
slid around and ready to start doing lookups (resolve the SBAddress from a
load address, then resolve the symbol context).

So it really depends on where you are getting the addresses that you want
to symbolicate. If you get a backtrace, wait for a while, then attach to a
process so you can symbolicate, the process might have loaded/unloaded one
or more shared libraries in the time gap since the backtrace was taken and
there are no guarantees that the addresses will resolve to the right
symbols.

If you are trying to use LLDB within your process, you won't be able to
attach to yourself. We also don't have a mode where LLDB attaches for
viewing only (not for debugging).

Is there any plan to support that? Or is that not needed?

This kind of viewing only attach would then allow LLDB to attach to the
process in which it is running. If you want to use LLDB for this kind of
scenario, you will need to save the stack backtraces and also find a way to
save all of the shared library load locations so you can later symbolicate
with a separate process.

In my case, I want to be able to symbolicate (with lldb) stacktrace of my
currently running process while the process is running (ie not after it
died), and with minimal overhead. Are you sure there is no way to achieve
that?

For example, I'm thinking of using dl_iterate_phdr(callback, context):
foreach section in module.GetSections
addr_t start = (addr_t)info->dlpi_addr + sect.GetFileAddress();
ctx->target.SetSectionLoadAddress(sect, start);

eg: something similar to here:
http://data-race-test.googlecode.com/svn-history/r3966/trunk/v2/tsan/tsan_symbolize_lldb_linux.cc

Could something like that work?

Otherwise :
* launching from A a process B that attaches to A and then computes
symbolication of A's backtrace and returning it to A is what I did, but
this is too slow.

* Related question: can we launch a new thread instead of starting a new
process to achieve this?

Greg_Clayton1 · September 24, 2013, 9:00pm

If you do follow my steps 1 - 4, you just need to be able to set the section load locations correctly. "info->dlpi_addr" will either be a slide, or an absolute address which you can then use in either the slide or section setting code. If you read up on the dl_iterate_phdr() function you should be able to determine which it is.

So yes, you should be able to do this as long as you can get the section load addresses for each module.

Greg

Yin_Ma · September 26, 2013, 12:34am

Hello,

What I want is to run a command and return to lldb only after
This command finishes. This should be the behavior of
synchronous mode. So I called
debugger.SetAsync(False)
right after lldb starts. However, I never get what
I expected. I set breakpoint and run. The process
Never stops, it just shows process exited.

Based on the webpage, it says:
When we step or continue, don't return from the function
Until the process stops.
So, when process stops at breakpoints, it should return
To lldb. Now it doesn't.

Is this the right behavior? How I can get the behavior
I want?

Thanks,

Yin

jingham · September 26, 2013, 12:47am

Can you send the example that isn't working, it's hard to reason about this in the abstract?

Jim

Enrico_Granata · September 26, 2013, 12:47am

First of all the obvious
Have you checked that your breakpoint is getting hit at all?
For instance, did you manually run lldb, set the breakpoint, and then run, and watch LLDB actually stop there?
If not, it seems a worthy initial test

If that works, more repro steps might be necessary to figure this out.
What inferior are you using? What does your script do exactly? That kind of information

Enrico Granata
egranata@.com
27683

Yin_Ma · September 26, 2013, 12:59am

Hi Enrico and Jim,

My lldb is 179.5 on mountain lion

What I did for the good result

File “xxx”

B main

R

Program stops at breakpoint and print thread and frame information.

For bad result

File “xxx”

B main

Command script import a.py

R

Process 10786 launched: ‘xxx’ (x86_64)

In the a.py

Def __lldb_init_module (debugger, dict):

Debugger.SetAsync(False)

This is about what I did.

Thanks,

Yin