Use lldb.so to create stack trace

Hi folks,

We are looking for a way to use lldb.so to print out a stack trace programmatically. In our scientific simulation tool (written in C++) , we use a custom terminate function where we are already able to print stack traces in case of an abnormal abort, which uses some gcc-specific magic.

My idea is to link our tool to lldb.so when compiling for debugging and to use the library to print a stack trace programmatically, including demangled names and correct line numbers. However, I don't really know how to start thus I am wondering

- has someone here done or seen something like this before?
- is there a tutorial for linking to lldb.so and making use of it through C++?
- would this also work when compiling with other compilers than clang?

Any help or pointers to relevant projects/documentation etc is greatly appreciated!

Regards,

Michael

Instead of linking lldb.so, which is basically an entire debugger, maybe you could just link llvm? I don’t remember which library has the stack trace function in llvm, but the function I believe is called llvm::sys::printStackTrace(). This will give you a much smaller overhead than bringing in all of lldb unnecessarily.

That would work for me too - I just did not know that there is such a capability in LLVM itself. I believe I found the printStackTrace method to be in libLLVMSupport.a. Would you happen to know what I have to do (or where to find more help) to make use of it?

Michael

PS: Maybe this is getting too OT; if yes, please let me know.

#include “llvm/Support/Process.h” // I think it’s in process.h, going from memory here.

void foo()
{
llvm::sys::printStackTrace(llvm::outs); // print to stdout
}

If you want to write it to a std::string instead you can use an llvm::raw_string_ostream.

If you need more help I can do so tomorrow or Wednesday when I’m back in the office. The above should work across multiple platforms though (Windows, Mac, Linux) and be fully symbolized (if your program is built with debug information)

This works if all you want to do is print the stack trace.

However, if you wish to examine or modify the trace in ANY WAY, then LLVM’s printStackTrace() is no longer sufficient.

In my case, my language mangles names in its own way, because it is not C++. I need to demangle these names, and therefore need to examine/modify the output.

Right now I am forking, launching LLDB in the parent, and attaching to my compiler in the child, and when LLDB gets notified of a signal, I print the stack trace. This is essentially what the OP wanted. This works, but there is a pause at startup while LLDB does??? (I think it’s loading symbols). I’d like to get rid of that pause, but have found no workable way of doing so.

One idea I had was to have LLDB launch only after my signal handler has caught a signal. Alas, in this mode, LLDB could not backtrace past the signal handler - all stack frames from my compiled code were dropped, so that it looked like the code that generated the signal was in fact my runtime.

Hi David,

I am interested about what you can achieve with your approach because we are planing to do something similar for Android in the future (not too soon).

Based on my understanding there are a few issues you have to handle in your current setup. First of all unwinding in LLDB isn’t as reliable as it should be especially if we don’t have full debug info. This can cause some flakiness in your output and can explain why you don’t see the backtrace past the signal handler. There are some work going on to improve the general unwinding code but there are a lot of outstanding edge cases. The pause at startup time is most likely caused by symbol loading and dwraf parsing. Currently LLDB try to do it lazily but it isn’t working as expected and a lot of information is loaded what you won’t use for your limited feature set. You can try to optimize these issues in LLDB but I am not sure if you can get the required performance. (Depending on which version of LLDB you use you might also spawn an lldb-server instance when doing the attach.)

I think the best solution would be to write a library similar to libunwind based on LLDB and put it into the LLDB codebase so it can rely on the internal data structures because currently that is the only way you can get rid of all performance hit imposed by the other parts of LLDB what you don’t use. It is a bigger task but if you would like to go this way then let me know and I am happy to discuss it.

Tamas

Hi Zachary,

I tried to compile the following MWE

#include “llvm/Support/Signals.h”
#define __STDC_LIMIT_MACROS
#define __STDC_CONSTANT_MACROS
#include “llvm/Support/raw_ostream.h”

int main() {
llvm::sys::PrintStackTrace(llvm::errs());
}

using the following command:

clang++ -std=c++11 -stdlib=libc++ -I/pds/opt/llvm/include mwe.cpp -o main -lc++ -lc++abi /pds/opt/llvm/lib/libLLVMSupport.a

This throws an error about missing the pthread library, so I add “-lpthread” to the end of the above line. However, this results in many, may errors like this:

/pds/opt/llvm/lib/libLLVMSupport.a(Debug.cpp.o): In function llvm::isCurrentDebugType(char const*)': /pds/opt/install/llvm/llvm-20150720-r242680-src/lib/Support/Debug.cpp:(.text._ZN4llvm18isCurrentDebugTypeEPKc+0xd7): undefined reference to std::string::compare(char const*) const’

/pds/opt/llvm/lib/libLLVMSupport.a(Triple.cpp.o): In function llvm::Triple::getLittleEndianArchVariant() const': /pds/opt/install/llvm/llvm-20150720-r242680-src/lib/Support/Triple.cpp:(.text._ZNK4llvm6Triple26getLittleEndianArchVariantEv+0xb): undefined reference to std::basic_string<char, std::char_traits, std::allocator >::basic_string(std::string const&)’

Any idea what I’m doing wrong?

Regards,

Michael

Unfortunately I’m not too sure. Neither clang nor linux are my primary development environments, so you probably need someone else to help you with this.

I’ve added someone to the thread though who is probably more familiar with this than I am, or who can at least refer you to the right person.

That said, If I had to take a wild guess, it looks like the errors you’re getting indicate that something is wrong with the way you’re linking against the C++ standard library. Maybe you’re pointing to a different version of the standard library than what you built LLVM against?

Hi folks,

We are looking for a way to use lldb.so to print out a stack trace programmatically. In our scientific simulation tool (written in C++) , we use a custom terminate function where we are already able to print stack traces in case of an abnormal abort, which uses some gcc-specific magic.

My idea is to link our tool to lldb.so when compiling for debugging and to use the library to print a stack trace programmatically, including demangled names and correct line numbers. However, I don't really know how to start thus I am wondering

- has someone here done or seen something like this before?

Not sure who has done anything like this.

- is there a tutorial for linking to lldb.so and making use of it through C++?

At the end of the email is an example of something that will attach to its parent process and iterate through all of the threads/frames.

- would this also work when compiling with other compilers than clang?

If the debug info is DWARF, then yes.

Any help or pointers to relevant projects/documentation etc is greatly appreciated!

Let me know if this helps.

Code sample:

#include <stdint.h>
#include <stdlib.h>

#if defined(__APPLE__)
#include <LLDB/LLDB.h>
#else
#include "LLDB/SBBlock.h"
#include "LLDB/SBCompileUnit.h"
#include "LLDB/SBDebugger.h"
#include "LLDB/SBFrame.h"
#include "LLDB/SBFunction.h"
#include "LLDB/SBModule.h"
#include "LLDB/SBStream.h"
#include "LLDB/SBSymbol.h"
#include "LLDB/SBTarget.h"
#include "LLDB/SBThread.h"
#include "LLDB/SBProcess.h"
#endif

#include <string>

using namespace lldb;

int
main (int argc, char const *argv)
{
    // Use a sentry object to properly initialize/terminate LLDB.
    SBDebugger::Initialize();
    
    SBDebugger debugger (SBDebugger::Create());
    
    // Create a debugger instance so we can create a target
    if (!debugger.IsValid())
    {
        fprintf (stderr, "error: failed to create a debugger object\n");
        exit(1);
    }
    
    SBError error;
    
    pid_t pid = getppid(); // Assuming parent process is what we want to backtrace
    SBAttachInfo attach_info(pid);
    
    const char *filename = nullptr;
    const char *target_triple = nullptr;
    const char *platform_name = nullptr;
    bool add_dependent_modules = false;
    SBTarget target = debugger.CreateTarget (filename, target_triple, platform_name, add_dependent_modules, error);
    
    SBStream strm;
    const bool append = false;
    strm.RedirectToFile ("/tmp/stack.txt", append);
    if (target.IsValid())
    {
        SBProcess process = target.Attach (attach_info, error);
        if (process.IsValid())
        {
            // Get the description of the process
            process.GetDescription(strm);
            
            const uint32_t num_threads = process.GetNumThreads();
            for (uint32_t thread_idx = 0; thread_idx < num_threads; ++thread_idx)
            {
                lldb::SBThread thread = process.GetThreadAtIndex (thread_idx);
                if (thread.IsValid())
                {
                    // Do one of the following:
                    
                    // 1 - Get a description of the thread
                    thread.GetDescription(strm);
                    // 1 - END
                    
                    // 2 - Do things yourself by using the public API to look a the stop reason
                    lldb::StopReason thread_stop_reason = thread.GetStopReason();
                    switch (thread_stop_reason)
                    {
                    case eStopReasonInvalid:
                    case eStopReasonNone: // Thread is stopped because other threads were stopped
                    case eStopReasonTrace: // Thread completed a single instruction step
                        break;
                    case eStopReasonBreakpoint:
                        {
                            uint32_t idx = 0;
                            uint64_t bp_id, bp_loc_id;
                            do
                            {
                                bp_id = thread.GetStopReasonDataAtIndex(idx++);
                                bp_loc_id = thread.GetStopReasonDataAtIndex(idx++);
                                SBBreakpoint bp = target.FindBreakpointByID(bp_id);
                                SBBreakpointLocation bp_loc = bp.FindLocationByID (bp_loc_id);
                                // Do something with "bp" if desired
                            } while (bp_id != 0);
                        }
                        break;
                    case eStopReasonWatchpoint:
                        {
                            const uint64_t wp_id = thread.GetStopReasonDataAtIndex(0);
                            SBWatchpoint wp = target.FindWatchpointByID(wp_id);
                            // Do something with "wp" if desired
                        }
                        break;
                    case eStopReasonSignal:
                        {
                            const uint64_t signo = thread.GetStopReasonDataAtIndex(0);
                            // Do something with "signo" if desired
                        }
                        break;
                        
                    case eStopReasonException:
                    case eStopReasonExec:
                    case eStopReasonPlanComplete:
                    case eStopReasonThreadExiting:
                    case eStopReasonInstrumentation:
                        break;
                    }
                    
                    char decription[1024];
                    
                    if (thread.GetStopDescription(description, sizeof(decription)))
                    {
                        // Do something with the stop description
                    }
                    // 2 - END
                    
                    // Now iterate through all of the frames in the thread
                    const uint32_t num_frames = thread.GetNumFrames();
                    for (uint32_t frame_idx = 0; frame_idx < num_frames; ++frame_idx)
                    {
                        SBFrame frame = thread.GetFrameAtIndex (frame_idx);
                        if (frame.IsValid())
                        {
                            // Do one of the following:
                            
                            // 1 - Let the frame describe itself
                            frame.GetDescription (strm);
                            // 1 - END
                            
                            // 2 - Do things manually using the SB API
                            lldb::SBModule module = frame.GetModule();
                            lldb::SBCompileUnit compile_unit = frame.GetCompileUnit();
                            lldb::SBFunction function = frame.GetFunction();
                            lldb::SBBlock block = frame.GetBlock();
                            lldb::SBLineEntry line_entry = frame.GetLineEntry();
                            lldb::SBSymbol symbol = frame.GetSymbol();
                            
                            // Now use the above objects
                            
                            // 2 - END
                        }
                    }
                }
            }
            
            process.Detach();
        }
    }
    
    SBDebugger::Terminate();

    return 0;
}

Linking is order-dependent:

clang++ -std=c++11 -stdlib=libc++ -I/pds/opt/llvm/include mwe.cpp -o main -lc++ -lc++abi /pds/opt/llvm/lib/libLLVMSupport.a

You are linking in LLVMSupport.a after libstdc++, so libstdc++ won’t be used to resolve any symbols from LLVMSupport.a.

Interchange the order of the linking, putting -lc++ at the end ought to work.

This seems to work now, thanks. For my MWE I get nicely formatted stack traces.

However, one problem remains: if I try to link my actual application using virtually the same libraries on the command line, it fails with the following error:

/usr/lib64/gcc/x86_64-suse-linux/4.9/…/…/…/…/x86_64-suse-linux/bin/ld: /pds/opt/llvm/lib/libLLVMSupport.a(Path.cpp.o): undefined reference to symbol ‘_ZNSs4_Rep10_M_destroyERKSaIcE@@GLIBCXX_3.4
/usr/lib64/libstdc++.so.6: error adding symbols: DSO missing from command line

Selectively removing or adding libraries at the end of the linker command does not help. Any ideas what might be wrong here?

Michael

PS: this is the full command minus some object files omitted for clarity (llvm lives in /pds/opt/llvm):

/pds/opt/llvm/bin/clang++ -O0 -g3 -fno-inline -DCOMPILER_ATTRIBUTES -std=c++11 -stdlib=libc++ -Wall -Wextra -pedantic -Wshadow -Wfloat-equal -Wcast-align -Wfloat-equal -Wdisabled-optimization -Wformat=2 -Winvalid-pch -Winit-self -Wmissing-include-dirs -Wredundant-decls -Wpacked -Wpointer-arith -Wstack-protector -Wswitch-default -Wwrite-strings -Wno-type-safety -Werror -Wunused -Wno-infinite-recursion -Wl,-rpath,/pds/opt/openmpi/lib64:/pds/opt/PointwiseV17.0R2/linux_x86_64/plugins:/pds/opt/intel/composer_xe_2015/lib/intel64/:/pds/opt/llvm/lib:/pds/opt/StarCD/lib:/pds/opt/torque/lib src/CMakeFiles/zfs.dir/zfscartesiangrid_inst_fv.cpp.o src/CMakeFiles/zfs.dir/main.cpp.o -o src/zfs -rdynamic /pds/opt/netcdf/lib64/libnetcdf.a /pds/opt/fftw/lib64/libfftw3.a /pds/opt/parallel-netcdf/lib/libpnetcdf.a /pds/opt/openmpi/lib64/libmpi_cxx.so /pds/opt/openmpi/lib64/libmpi.so /pds/opt/llvm/lib/libLLVMSupport.a -lrt -ldl -ltinfo -lpthread -lz -lm /pds/opt/llvm/lib/libc++abi.a -lpthread -lz -lm /pds/opt/llvm/lib/libc++abi.a -Wl,-rpath,/pds/opt/openmpi/lib64