lldb: how can I make it reliably scriptable?

Sorry for the length here. TL;DR: is it possible to reliably script
LLDB non-interactively, or is this a known area of weakness in LLDB and
I should give up until LLDB gets more mature in this area?

We have a script in our test environment to perform a basic triage of
coredumps that happen during testing; when a test detects a core it runs
this script to get some info about the failure: we can use the backtrace
to categorize the core, etc.

On GNU/Linux it uses GDB to do some simple things like print some
important global variable values, show a full backtrace of all threads
(these programs often have 20+ threads active), show the environment,
etc. We give this little script to GDB with the "-batch -x <file>"
commands, and it works perfectly every time.

Now we need to do the same thing for cores generated on Mac OSX (we're
using Xcode 5.1.1 but we've seen these same behaviors for older versions
as well).

At first we were using the old GDB 6.3.50.20050815-cvs which is the last
one supported by Apple. This works fine if we have a full set of object
files, but if we use dsymutil to generate .dSYM and DON'T have the
object files, this version of GDB can't give us full backtraces; we get
the function names only: no line numbers, no arguments, no ability to
access global variables, etc. Apparently it can't get full details from
the dSYM files. However, at least it does what little it does reliably.

So we first tried to get the latest GDB with homebrew and use that, but
it simply does not work at all:

  $ gdb -c core.78225 dist/bin/myprog
  GNU gdb (GDB) 7.8.1
    ..
  "/Users/build/crash/core.78225": no core file handler recognizes format

Building GDB myself from source gives the same result. So, we turned to
LLDB to try to get the same behavior... and it just does not work
reliably.

First, is there any sort of comprehensive manual for the LLDB CLI? On
lldb.llvm.org I see a tutorial and a command comparison with GDB, but
nowhere can I find any sort of manual akin to the GDB manual that
describes all the LLDB CLI commands, what they do, etc... ?

We tried to use the LLDB -s option to provide a script file; very
simple:

  $ cat show.lldb
  target create --core core.78225 dist/bin/myprog
  thread backtrace all
  exit

Now, with some core files this works just fine. However with other
cores, it fails completely:

  $ lldb --version
  lldb-310.2.37

  $ lldb -s show.lldb
    ..
  (lldb) thread backtrace all
  error: Aborting reading of commands after command #1: 'thread backtrace all' failed with error: invalid thread
  Aborting after_file command execution, command file: 'show.lldb' failed.

If I do it interactively, exact same commands, exact same core, etc.
rather than via -s then it works every time. But using -s it fails (for
some cores), every time. Using "-o" multiple times to pass individual
commands instead of "-s" fails the same way.

Even more frustrating is that when we get the above error, LLDB stops
processing the script file and SITS AT THE PROMPT because it never
processes the "exit" command!! This means our entire test suite hangs
right there. Redirecting stdin from /dev/null doesn't fix this; it's
even worse (it prints out 100's of "quit" commands then hangs).
Obviously not acceptable. There doesn't seem to be any LLDB equivalent
of GDB's "-batch" flag.

So then we gave up on the LLDB front-end and tried to write a Python
script to do the debugging that we want. Even though we just want to
run a couple of simple commands, this took some effort for my colleague
to learn the Python API and ended up with 160 lines of Python, but he
got it working.

Sometimes it works, I should say.

Other times, Python itself aborts ("Abort trap: 6"). Even this is not
so bad, because we can just put the invocation of Python into a loop and
eventually it usually works (unlike the -s option to lldb above, this
one doesn't depend on the core file; the same core file will sometimes
have Python dump core and sometimes not).

However, other times the Python script just hangs forever and never
finishes; again this means the entire automated test suite hangs in this
situation.

At this point I don't feel like we have any alternative except to tell
people that there's no support for this capability on OSX and they'll
just have to debug all the cores by hand interactively, and we can't do
any automated categorization of cores based on backtraces, etc.

I'm checking the output OSX dumps into ~/Library/Logs/DiagnosticReports
which doesn't have everything we need but at least has a stack trace.

Hi Paul,
it’s hard to tell exactly what is going on in your situation without further details, but

a) is there any chance you can compile LLDB from sources and use that debugger to perform your automation?
The LLDB you would get that way is newer than the one you are trying to use right now (and we fix bugs all the time, so you do want as new as you can get)

Also,
b) if the debugger hangs or crashes, it’s a bug and it should be fixed - so, please file bugs for anything that does not work
And, again, if you are somewhat tracking our trunk, any and all bug fixes, you get much more rapidly than staying on released Xcode versions

c) Without further details (crash logs, samples, …) it’s hard to say what’s going on in your situation, so not much to comment here - except that LLDB should be mature enough to script (not via the command line, but using the SB API) - and anything that blocks that should be filed on the LLVM bugzilla as an issue

it’s hard to tell exactly what is going on in your situation without
further details, but

I realize this is frustrating, but it's not a simple situation since the
the Python scripting version happens at random times; the Python crash
is moderately reproducible (1 in 10 or so) but the hang is relatively
rare... we never hit it during testing and it only happened after we put
the new facility into production and we started to see hangs.

a) is there any chance you can compile LLDB from sources and use that
debugger to perform your automation?
The LLDB you would get that way is newer than the one you are trying
to use right now (and we fix bugs all the time, so you do want as new
as you can get)

I could but it's just really tough. I could easily see if the "lldb -s"
bug is fixed since that's 100% reproducible if you have the right
(wrong?) kind of core file.

We have a suite of OSX build servers (managed by Bamboo) so I'd have to
deploy new LLDB to all of them, plus as a development team we'd have to
agree to accept some hanging/unreliable OSX build server results for a
while in order to try to track down this LLDB issue. That's a tough
sell.

Also,
b) if the debugger hangs or crashes, it’s a bug and it should be fixed
- so, please file bugs for anything that does not work
And, again, if you are somewhat tracking our trunk, any and all bug
fixes, you get much more rapidly than staying on released Xcode
versions

The problem is that it's Python that crashes, and no core is generated
(that I can see). I'm not sure why not, or how to get more information
about these crashes. Also this is using the /usr/bin/python (2.7.5)
that comes with OSX. We have our own locally-built Python 2.7.8 and if
I use THAT it dumps core every single time. Not sure if this is valid.

c) Without further details (crash logs, samples, …) it’s hard to say
what’s going on in your situation, so not much to comment here -
except that LLDB should be mature enough to script (not via the
command line, but using the SB API) - and anything that blocks that
should be filed on the LLVM bugzilla as an issue

I just don't think that I have enough information to create useful
bugzilla entries, and I don't know if I have enough tuits to dig into
this enough to get that information.

OK. I'll see if I can free up any time to dig into this more deeply.

This does not actually solve the underlying problem, but if all you want is to execute a bunch of lldb commands, you could try passing them in with expect (http://linux.die.net/man/1/expect). Lldb should not be able to tell the difference from you typing them in, so that would hopefully work. But do file bugs about anything you find not working. :slight_smile:

regards,
pl

Is there any way to build the latest LLDB on OSX from the command line?
I don't have an OSX system in front of me, I just ssh into one of the
Mac systems in our build/test rack. The build instructions seem to
assume I have a full graphical Xcode available.

Is there an xcodebuild invocation I can use from the CLI to build LLDB,
instead? I tried a few things but wasn't successful (I'm no guru when
it comes to Xcode, in any form).

Cheers!

CMake / ninja is supposed to work, but you may have to tinker around to find the correct set of flags to pass. To use it you would follow the instructions for CMake / ninja of some other platform, like linux.

The way I usually do things is

$ xcodebuild -scheme lldb-tool -configuration Debug

Your mileage may vary depending on your OS X/Xcode version (sometimes Xcode needs to have the license approved and to be authorized to do developer-y things the first time through, and there is a magic command-line incantation to do that without UI access but I can’t recall it)

My steps require one GUI session, then are command line. Can you use Chrome Remote Desktop or ?

One time setup:

Xcode Menu

Preferences Menu Item (⌘,)

Locations Tab

Derive Data: Relative

https://llvm.org/svn/llvm-project/lldb/trunk/docs/code-signing.txt

sudo xcodebuild -license

sudo dscl . append /Groups/_developer GroupMembership

sudo /opt/local/bin/port install swig swig-python ninja php

sudo ln -s /opt/local/bin/swig

sudo ln -s /opt/local/bin/swig /usr/bin/swig

Sync & build:

#!/bin/bash -ex

ROOT_DIR=$HOME/ll/tot

LLDB_CONFIG=Debug

LLDB_BIN=$ROOT_DIR/lldb/DerivedData/lldb/Build/Products/$LLDB_CONFIG

PYTHONPATH="$LLDB_BIN/LLDB.framework/Versions/A/Resources/Python"

DOTEST_OPTS="-m --executable $LLDB_BIN/lldb --framework $LLDB_BIN/LLDB.framework -A x86_64 -C clang -s $ROOT_DIR/lldb/DerivedData/lldb-test-results"

mkdir -p $ROOT_DIR

cd $ROOT_DIR

git clone http://llvm.org/git/lldb.git &

git clone http://llvm.org/git/llvm.git &

git clone http://llvm.org/git/clang.git &

wait

ln -s $ROOT_DIR/llvm lldb/llvm

ln -s $ROOT_DIR/clang llvm/tools/clang

first xcode build fails but second one will succeed!

xcodebuild -scheme lldb-tool -workspace $ROOT_DIR/lldb/lldb.xcworkspace -configuration $LLDB_CONFIG build || xcodebuild -scheme lldb-tool -workspace $ROOT_DIR/lldb/lldb.xcworkspace -configuration $LLDB_CONFIG build

cd $ROOT_DIR/lldb/test

./dosep.py --options “$DOTEST_OPTS”

The xcodebuild is pretty simple, you just check out the sources, cd to the top level of lldb and do:

$ xcodebuild -configuration Debug

or if you want a debug version of Clang as well use "DebugClang" or if you want a release version "Release".

However, either way you try (cmake or xcodebuild) you're going to have some problems building a debugserver that the system will trust. Building a code-signed debugserver is not that hard. You have to make the cert (as described in lldb/doc/code-signing.txt.) I'm sure there's some command-line way to do this, but I don't know what it is. This is a one-time thing however, so you can use Apple Remote Desktop to do that, and then do all your builds in the ssh session. The next bit is that to use the cert for code signing the build process will have to have access to your login keychain, which is unlocked by default for a GUI session, but not an ssh session. That's easy, just do:

$ security unlock-keychain

before building.

The trickier part is that there's really no good way to authenticate a hand-built debugserver from an ssh session. Normally, the first time you try to use a hand-built debugserver, it will issue the request to debug, which will go to the kernel, and that will ask out to some daemon for authentication, which will put up an authorization dialog. There's no way to do that in an ssh session. There are API's to enter the authorization manually on the command line, but this has to happen in debugserver, which has no access to a terminal.

However, there are special rules for Apple codesigned debugservers. If you just use Xcode to debug something once from a GUI session on the machine, it will do all the necessary setup for you. Again, this is a one-time thing, so you can do this once with ARD and then you won't need to do it again till you reinstall your OS... So it may be simpler to build lldb, then copy the debugserver from your Xcode installation (it's in Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources) to your lldb build.

Jim

Can you post the python file you are using to do the core file inspection? We might be able to spot some things that might be causing problems. And if you aren't using the lldb::SB API (i.e are you just doing a bunch of "SBDebugger::HandleCommand()" calls?), we might be able to help you get that part going. If you can make a core file available that fails with your script we can help figure out what is going wrong.

Greg Clayton

Here's the script, FWIW. Note that I'm not sure if it's any particular
core that causes the problem; when I run this script on the same core
multiple times sometimes it works, sometimes Python aborts, and
sometimes Python hangs.

Of course it could well be something about my particular program that
triggers this and others wouldn't. I'll see if I can provide a core
(presumably you'd need the executable as well... what about dSYM
content?)

We invoke this script with, basically:

  PYTHONPATH=$(lldb -P) python corescript.py core.12345 bin/myprog

#---- %>< snip ><% ----
#!/usr/bin/env python

import lldb
import sys

def load_core(debugger, exe, core):
    """

    @param exe: Path to the executable binary
    @param core: Path to the core
    @rtype: SBProcess
    @return:
    """
    target = debugger.CreateTargetWithFileAndArch(exe, lldb.LLDB_ARCH_DEFAULT)
    """ @type : SBTarget """
    if target:
        process = target.LoadCore(core)
        """ @type : SBProcess """
        if process:
            return process

        raise Exception("Could not load core")
    raise Exception("Could not create target")

def print_version(process):
    """
    @type process: SBProcess
    """
    thread = process.GetSelectedThread()
    """ @type : SBThread """
    if thread:
        frame = thread.GetSelectedFrame()
        """ @type : SBFrame """
        if frame:
            version = frame.EvaluateExpression('globals->version.data')
            """ @type : SBValue """
            if version:
                err = lldb.SBError()
                version_string = process.ReadCStringFromMemory(int(version.GetValue(), 16), 512, err)
                if err.Success():
                    print("*** Version: " + version_string)
                else:
                    print("*** Error extracting version")
                print("")
                return

            raise Exception("Invalid expression result, can't extract version")
        raise Exception("Invalid frame, can't extract version")
    raise Exception("Invalid thread, can't extract version")

def print_all_backtraces(process):
    """
    @type process: SBProcess
    """
    print("*** All Threads Backtrace:")
    for thread in reversed(process.threads):
        if thread:
            print_backtrace(thread)
        else:
            print("ERROR: Failed to print backtrace for a thread")

def print_backtrace(thread, include_label=False):
    """
    @type thread: SBThread
    """
    if include_label:
        print("*** Backtrace:")

    print("Thread %d (core thread %d):" % (thread.GetIndexID(), thread.GetThreadID()))
    for frame in thread:
        if frame:
            print(frame)
        else:
            print("ERROR: Failed to print a frame")

    print("")

def print_environment(process):
    """
    @type process: SBProcess
    """
    print("*** Environment:")
    print(' ' + '\n '.join(read_null_term_array(process, 'globals->environment->envp')))
    print("")

def read_null_term_array(process, expr, max_length=1000):
    """
    @type process: SBProcess
    """
    thread = process.GetSelectedThread()
    """ @type : SBThread """
    if thread:
        frame = thread.GetSelectedFrame()
        """ @type : SBFrame """
        if frame:
            array_value = frame.EvaluateExpression(expr)
            """ @type : SBValue """
            if array_value:
                array_address = array_value.GetValueAsUnsigned()
                idx = 0
                ptr_err = lldb.SBError()
                array_data = []
                for i in xrange(0, max_length):
                    element_ptr = process.ReadPointerFromMemory(array_address + idx * array_value.GetByteSize(),
                                                                ptr_err)
                    if ptr_err.Success():
                        if element_ptr == 0:
                            break

                        val_err = lldb.SBError()
                        element = process.ReadCStringFromMemory(element_ptr, 1024*10, val_err)
                        if val_err.Success():
                            array_data.append(element)
                        else:
                            print("ERROR: Unable to read value at address %s" % (str(element_ptr)))

                        idx += 1
                    else:
                        print("ERROR: Unable to read pointer at address %s" %
                              (str(array_address + idx * array_value.GetByteSize()),))
                        break

                return array_data

            raise Exception("Unable to read array address for %s" % (expr,))
        raise Exception("Invalid frame for %s" % (expr,))
    raise Exception("Invalid thread for %s" % (expr,))

def main():
    if len(sys.argv) != 3:
        print("usage: python %s core_file exe_file" % (sys.argv[0],))
        exit(1)

    core = sys.argv[1]
    exe = sys.argv[2]

    debugger = lldb.SBDebugger.Create()
    """ @type : SBDebugger """

    process = load_core(debugger, exe, core)
    if process:
        print_version(process)
        print_backtrace(process.GetSelectedThread(), True)
        print_all_backtraces(process)
        print_environment(process)
    else:
        print("Failed to debug core " + core)

if __name__ == '__main__':
    main()

I found the race condition. Seems Process::LoadCore() doesn't ensure the target is stopped before it returns:

Crashed Thread: 7

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: EXC_I386_GPFLT

Thread 0:: Dispatch queue: com.apple.main-thread
0 _lldb.so 0x000000010df4d161 std::__1::weak_ptr<lldb_private::Process>::lock() const + 1
1 _lldb.so 0x000000010fcc1d63 lldb_private::UnwindLLDB::DoGetFrameInfoAtIndex(unsigned int, unsigned long long&, unsigned long long&) + 75
2 _lldb.so 0x000000010fd88c8b lldb_private::Unwind::GetFrameInfoAtIndex(unsigned int, unsigned long long&, unsigned long long&) + 61
3 _lldb.so 0x000000010fd87766 lldb_private::StackFrameList::GetFramesUpTo(unsigned int) + 286
4 _lldb.so 0x000000010fd882c1 lldb_private::StackFrameList::GetFrameAtIndex(unsigned int) + 185
5 _lldb.so 0x000000010fd63060 lldb_private::thread::GetSelectedFrame() + 48
6 _lldb.so 0x000000010df4008b lldb::SBThread::GetSelectedFrame() + 203
7 _lldb.so 0x000000010dff19e5 _wrap_SBThread_GetSelectedFrame(_object*, _object*) + 128
8 org.python.python 0x000000010ce7c7c9 PyEval_EvalFrameEx + 14387
9 org.python.python 0x000000010ce7f60e 0x10cdf5000 + 566798
10 org.python.python 0x000000010ce7c3e3 PyEval_EvalFrameEx + 13389
11 org.python.python 0x000000010ce7f60e 0x10cdf5000 + 566798
12 org.python.python 0x000000010ce7c3e3 PyEval_EvalFrameEx + 13389
13 org.python.python 0x000000010ce7f60e 0x10cdf5000 + 566798
14 org.python.python 0x000000010ce7c3e3 PyEval_EvalFrameEx + 13389
15 org.python.python 0x000000010ce78d62 PyEval_EvalCodeEx + 1413
16 org.python.python 0x000000010ce787d7 PyEval_EvalCode + 54
17 org.python.python 0x000000010ce987bd 0x10cdf5000 + 669629
18 org.python.python 0x000000010ce98860 PyRun_FileExFlags + 133
19 org.python.python 0x000000010ce983fd PyRun_SimpleFileExFlags + 769
20 org.python.python 0x000000010cea9b23 Py_Main + 3051
21 libdyld.dylib 0x00007fff87fa25c9 start + 1

Thread 7 Crashed:
0 _lldb.so 0x000000010df4d17c std::__1::weak_ptr<lldb_private::Process>::lock() const + 28
1 _lldb.so 0x000000010fcc1d63 lldb_private::UnwindLLDB::DoGetFrameInfoAtIndex(unsigned int, unsigned long long&, unsigned long long&) + 75
2 _lldb.so 0x000000010fd88c8b lldb_private::Unwind::GetFrameInfoAtIndex(unsigned int, unsigned long long&, unsigned long long&) + 61
3 _lldb.so 0x000000010fd8795e lldb_private::StackFrameList::GetFramesUpTo(unsigned int) + 790
4 _lldb.so 0x000000010fd87246 lldb_private::StackFrameList::ResetCurrentInlinedDepth() + 46
5 _lldb.so 0x000000010fda1bcf lldb_private::thread::ShouldStop(lldb_private::Event*) + 719
6 _lldb.so 0x000000010fda7790 lldb_private::ThreadList::ShouldStop(lldb_private::Event*) + 488
7 _lldb.so 0x000000010fd77f19 lldb_private::Process::ShouldBroadcastEvent(lldb_private::Event*) + 379
8 _lldb.so 0x000000010fd754c5 lldb_private::Process::HandlePrivateEvent(std::__1::shared_ptr<lldb_private::Event>&) + 365
9 _lldb.so 0x000000010fd7865b lldb_private::Process::RunPrivateStateThread() + 511
10 _lldb.so 0x000000010fd7810f lldb_private::Process::PrivateStateThread(void*) + 9
11 libsystem_pthread.dylib 0x00007fff81357268 _pthread_body + 131
12 libsystem_pthread.dylib 0x00007fff813571e5 _pthread_start + 176
13 libsystem_pthread.dylib 0x00007fff8135541d thread_start + 13

Thread 7 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000001 rbx: 0x000000012a2ce798 rcx: 0x0000000000000000 rdx: 0x0000000000000000
  rdi: 0x000000012a2ce798 rsi: 0x10000000000000f0 rbp: 0x000000012a2ce780 rsp: 0x000000012a2ce770
   r8: 0x0000000000000000 r9: 0x000000012a2cf000 r10: 0x00007f8fca82f400 r11: 0x00007f8fca82fdd0
  r12: 0x0000000000000000 r13: 0x00007f8fc951b9a0 r14: 0x10000000000000f0 r15: 0x000000012a2cea10
  rip: 0x000000010df4d17c rfl: 0x0000000000010206 cr2: 0x000000010fea75f8

The issue is we have a read/write lock that controls when stuff can be done to a process. Multiple threads can request that the run lock stay stopped, and only one can request that it runs. So the problem is we call lldb::SBThread::GetSelectedFrame() on the main thread while the private state thread is still finding out that the process has stopped. lldb::SBThread::GetSelectedFrame() tries to get the run lock and keep the process stopped, and it succeeds because LoadCore wasn't synchronously making sure the core file was ready to be played with.

So the solution is to make sure that SBTarget::LoadCore() is a synchronous call that has the target stopped _before_ it returns. Then the process will be setup and ready to allow requests.

A work around for now is to call time.sleep(1) after calling LoadCore() to give the Process::PrivateStateThread() time to get settled. Other race conditions could cause the run lock to no be acquired:

lldb::SBFrame
SBThread::GetSelectedFrame ()
{
    Log *log(lldb_private::GetLogIfAllCategoriesSet (LIBLLDB_LOG_API));

    SBFrame sb_frame;
    StackFrameSP frame_sp;
    Mutex::Locker api_locker;
    ExecutionContext exe_ctx (m_opaque_sp.get(), api_locker);

    if (exe_ctx.HasThreadScope())
    {
        Process::StopLocker stop_locker;
        if (stop_locker.TryLock(&exe_ctx.GetProcessPtr()->GetRunLock()))
        {
        }
        else
        {
            if (log)
                log->Printf ("SBThread(%p)::GetSelectedFrame() => error: process is running",
                             static_cast<void*>(exe_ctx.GetThreadPtr()));
        }

And an error would be returned and an empty SBFrame would be returned.

I'll have a fix committed shortly.

Greg

Fixed with:

% svn commit source/Target/Process.cpp
Sending source/Target/Process.cpp
Transmitting file data .
Committed revision 230060.

Please download the top of tree sources, build the Release build and make sure it works for you. Thanks of the info so that we could track down and fix this issue.

Greg Clayton

One more fix:

% svn commit source/Target/Process.cpp
Sending source/Target/Process.cpp
Transmitting file data .
Committed revision 230066.

This ensures if you ask the process for its state, that it will respond with eStateStopped.

Cool! I'll try the workaround and the latest version.

Question: would this impact only the Python interface? Or could it be
seen in the normal LLDB CLI interface as well? Also, could this result
in a hang, instead of a crash?

One of my scripts tries to use the LLDB -s scripting interface just to
list the image info, so I wrote a simple script:

   target create -c core.12345
   image list
   exit

However, when I run this from my script (using "lldb -s") it will
regularly hang (although it works OK interactively). I have to run it a
couple of times to get a good result.

Is there another/better way to get the fully-qualified name of the
executable that was running, given nothing but the coredump?

I found the race condition.

Cool! I'll try the workaround and the latest version.

Question: would this impact only the Python interface? Or could it be
seen in the normal LLDB CLI interface as well?

Probably if you source a LLDB command file you could run into this. Typing by hand is probably going to be ok.

Also, could this result
in a hang, instead of a crash?

Quite possibly. Any race condition that borks the heap can cause general mayhem.

One of my scripts tries to use the LLDB -s scripting interface just to
list the image info, so I wrote a simple script:

  target create -c core.12345
  image list
  exit

However, when I run this from my script (using "lldb -s") it will
regularly hang (although it works OK interactively). I have to run it a
couple of times to get a good result.

The race condition is definitely there for source command files. Since it may or may not bork the heap, I would suggest possibly adding a "script import time; time.usleep(250000)" before the image list unless you get the top of tree sources.

Is there another/better way to get the fully-qualified name of the
executable that was running, given nothing but the coredump?

No, the dynamic loader will be able to figure it out and then you might need to lookup your binary on a build server using the UUID, path and triple. So there is no better way than just loading the core dumper and letting the dynamic loader figure it out for real. Of course you might be able to improve your core dump to include the main executable somewhere in notes, or sections, but I am guessing this probably isn't a viable option for most.

Greg

The race condition is definitely there for source command files. Since
it may or may not bork the heap, I would suggest possibly adding a
"script import time; time.usleep(250000)" before the image list unless
you get the top of tree sources.

Ah. Clever. I'll give that a go.

I did build the latest code and as predicted it failed during codesign:

      /bin/sh -c /Users/build/pds/lldb/build/lldb.build/Release/lldb-server.build/Script-4C3326CA18B2A2B800EB5DD7.sh
  lldb_codesign: no identity found

I haven't had time to dig into this further yet.

> Is there another/better way to get the fully-qualified name of the
> executable that was running, given nothing but the coredump?

No, the dynamic loader will be able to figure it out and then you
might need to lookup your binary on a build server using the UUID,
path and triple.

Sure; in my case I'm examining the core immediately after it happens (we
check for cores after we run each test) so I always have the binary
right there... but tests can run multiple binaries and it's not always
obvious which one the core belongs to. Getting the fully-qualified path
out of the core file is all I need for this particular situation.

Cheers!

The race condition is definitely there for source command files. Since
it may or may not bork the heap, I would suggest possibly adding a
"script import time; time.usleep(250000)" before the image list unless
you get the top of tree sources.

Ah. Clever. I'll give that a go.

I did build the latest code and as predicted it failed during codesign:

     /bin/sh -c /Users/build/pds/lldb/build/lldb.build/Release/lldb-server.build/Script-4C3326CA18B2A2B800EB5DD7.sh
lldb_codesign: no identity found

I haven't had time to dig into this further yet.

Follow the instructions in:

svn cat http://llvm.org/svn/llvm-project/lldb/trunk/docs/code-signing.txt

Is there another/better way to get the fully-qualified name of the
executable that was running, given nothing but the coredump?

No, the dynamic loader will be able to figure it out and then you
might need to lookup your binary on a build server using the UUID,
path and triple.

Sure; in my case I'm examining the core immediately after it happens (we
check for cores after we run each test) so I always have the binary
right there... but tests can run multiple binaries and it's not always
obvious which one the core belongs to. Getting the fully-qualified path
out of the core file is all I need for this particular situation.

Gotcha, you will still need to load it in LLDB and let the dynamic loader figure it out.

On linux you could use file(1), and you could also tune
kernel.core_pattern to something like this "%E/%e.%p".

Or even better you could pipe core (man 5 core, in short --
kernel.core_pattern must starts with |, and be aware of quotes) into
user space program, that will save information about it somewhere else,
and using this aproach you don't need space for storing core.

But all of this is for linux only.

Cheers,
Azat.

Yes, on GNU/Linux I have this all sorted and it works fine. I'm trying
to get the same operations with our OSX cores.