Assertion failure in DYLDRendezvous::UpdateSOEntries()

I just compiled the latest LLDB HEAD on Linux 64-bit and am seeing an assertion failure on launch when debugging a specific application (Autodesk maya):

lldb: …/tools/lldb/source/Plugins/DynamicLoader/POSIX-DYLD/DYLDRendezvous.cpp:206: bool DYLDRendezvous::UpdateSOEntries(): Assertion `m_previous.state == eConsistent’ failed.

It’s been awhile since I’ve rebuilt from HEAD so not sure when this may have been introduced and simply commenting out the assertion appears to run fine.

Does anyone know if this assertion is still relevant and if so have any tips on where I should look to figure out what’s happening?

Thanks,
Andrew

I did a POSIX Dynamic Loader change yesterday. It might be causing this (not obvious yet, but could be.) r219371. If you’re in a position to do a test of your run with that one change reverted out, and if that fixes it, I can yank out that change.

Ideally I’d love to isolate if that is the cause of the changed behavior.

-Todd

Hi Todd,

I just backed out your change and the behaviour remained the same. When I say it’s been awhile since I rebuilt though the last build I have around is from June and it doesn’t assert there. Has the default assertion behaviour changed to enabled by default? I’m building with “cmake -G Ninja …”.

In fact checking the June build I don’t get an assertion failure (I assume they’re disabled) but the values do differ and I would get one if they were enabled. So it looks like this issue may have been around for awhile.

I can investigate further if you have any pointers around what I might want to look for.

Cheers,
Andrew

I just took a peek at the code around there. It’s not immediately clear whether the assert is entirely useful, but I think it is intending to guard against adding new dynamic library info when the previous state of the data structures are not settled.

I think a good starting point for you would be to enable ‘log enable lldb dyld’ and see what kind of activity is showing up for the existing log lines. Maybe post that here if that doesn’t help.

I probably wouldn’t remove the assert until we understand the root issue since the assert seems reasonable.

Worst case if this doesn’t get you further, you can add some if code around the condition and stick a breakpoint in the unexpected case so you can break the debugger in action and inspect what’s going on.

I hope that gets you started!

-Todd

I don’t see anything very interesting in the dyld log but it looks like we’re hitting the breakpoint at DynamicLoaderPOSIXDYLD::RendezvousBreakpointHit() where its rendezvous state is first eAdd and then hitting it again and the state is eDelete without transitioning to eConsistent between eAdd and eDelete. I’ve never looked at this runtime linker stuff before so I’m not sure if this is a valid transition or not, this is what’s causing the assertion failure though. Any idea?

Cheers,
Andrew

I’m not sure if this is a valid transition or not

I’m not sure yet either - can you try another scenario and see if you’re consistently hitting that? (I mean without Maya — are you trying to debug plugins?) If you are doing some kind of plugin that gets dynamically loaded, can you write a bare-bones exe that loads a bare-bones shared library (.so) manually and see if you hit the same issue on that? I’m just looking to help isolate the scenario.

The other thing that would be interesting would be to know if maybe there is an exec (essentially executing another exe) happening in between? If that was the case, there might be an issue where a clean sweep call to reset state isn’t doing the right thing. You might see more info on exec behavior with ‘log enable lldb process’.

Let me know how that goes.

I ran everything with LD_DEBUG=all to see what was being loaded around the assertion and tracked it down to a dlopen() on a library that exists but that has unmet dependencies of its own. Writing a trivial C program causes the same assertion failure:

#include <dlfcn.h>
int main()
{
dlopen(“./libqmng.so”, RTLD_NOW);
return 0;
}

Where “ldd libqmng.so” gives:

linux-vdso.so.1 => (0x00007fffe61fe000)
libmng.so.1 => not found

So it looks like this may be a valid transition on a failed dlopen() call?

Ok that’s starting to sound like something we probably wouldn’t cover.

So maybe it’s like when (in your example) the loader starts processing libqmng.so, does the add, then immediately reverses it out as it hadn’t officially “finished”, and then reverses it out.

So it’s probably fair to change that assert to asserting it is either stable, or if the previous state was an add (for the delete case). In any event, the assert as it stands seems not inclusive enough.

I’d be okay with removing that one since we’ve found a case where it doesn’t hold.

Ok great, if you’re ok with it then I will update that assert to be:

assert(m_previous.state == eConsistent || (m_previous.state == eAdd && m_current.state == eDelete));

since it would still be invalid (as far as we can tell) to be in there on a transition from eDelete to eAdd.

Let me know if that looks okay and I’ll push the change.

Looks good.