Segfault using the lldb python module with a non-Xcode python binary

I’m building lldb with python3 support by using the framework that is shipped with the latest versions of Xcode.

I’m able to build and run lldb just fine but if I try to use the lldb python module on a python binary that is not the one from Xcode it segfaults when creating the module. I then tried with the stock lldb from Xcode and found the exact same issue :frowning:. I don’t think this was a problem before?

I’m not sure why this happens and I wasn’t able to debug the issue. I’ve already tried with a binary that has the exact same version of python but still the same problem:

Works fine with the Xcode binary:

$ PYTHONPATH=lldb -P /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/bin/python3

Python 3.7.3 (default, Sep 18 2019, 14:29:06)

[Clang 11.0.0 (clang-1100.0.33.8)] on darwin

Type “help”, “copyright”, “credits” or “license” for more information.

import lldb

Fails with any other:

$ PYTHONPATH=lldb -P /Users/aadsm/.pyenv/versions/3.7.3/bin/python

Python 3.7.3 (default, Nov 12 2019, 23:19:54)

[Clang 11.0.0 (clang-1100.0.33.8)] on darwin

Type “help”, “copyright”, “credits” or “license” for more information.

import lldb

Segmentation fault: 11

I attached lldb to see where it was failing and it’s right after liblldb is loaded and python is trying to create the module itself, in the PyModule_Create2 function (https://github.com/python/cpython/blob/master/Objects/moduleobject.c#L173-L179).

The disassembly shows:

Process 89097 stopped

  • thread #1, queue = ‘com.apple.main-thread’, stop reason = EXC_BAD_ACCESS (code=1, address=0x10)

frame #0: 0x000000010f4cae5b Python3`PyModule_Create2 + 27

Python3`PyModule_Create2:

→ 0x10f4cae5b <+27>: movq 0x10(%rax), %rdi

0x10f4cae5f <+31>: callq 0x10f5823b0 ; _PyImport_IsInitialized

0x10f4cae64 <+36>: testl %eax, %eax

0x10f4cae66 <+38>: je 0x10f4cae77 ; <+55>

Target 0: (Python) stopped.

(lldb) dis

Python3`PyModule_Create2:

0x10f4cae40 <+0>: pushq %rbp

0x10f4cae41 <+1>: movq %rsp, %rbp

0x10f4cae44 <+4>: pushq %r14

0x10f4cae46 <+6>: pushq %rbx

0x10f4cae47 <+7>: movl %esi, %r14d

0x10f4cae4a <+10>: movq %rdi, %rbx

0x10f4cae4d <+13>: leaq 0x2226ac(%rip), %rax ; _PyRuntime

0x10f4cae54 <+20>: movq 0x5a0(%rax), %rax

→ 0x10f4cae5b <+27>: movq 0x10(%rax), %rdi

0x10f4cae5f <+31>: callq 0x10f5823b0 ; _PyImport_IsInitialized

0x10f4cae64 <+36>: testl %eax, %eax

0x10f4cae66 <+38>: je 0x10f4cae77 ; <+55>

0x10f4cae68 <+40>: movq %rbx, %rdi

0x10f4cae6b <+43>: movl %r14d, %esi

0x10f4cae6e <+46>: popq %rbx

0x10f4cae6f <+47>: popq %r14

0x10f4cae71 <+49>: popq %rbp

0x10f4cae72 <+50>: jmp 0x10f4cae90 ; _PyModule_CreateInitialized

0x10f4cae77 <+55>: leaq 0x14f111(%rip), %rdi ; “Python import machinery not initialized”

0x10f4cae7e <+62>: callq 0x10f593d40 ; Py_FatalError

0x10f4cae83 <+67>: nopw %cs:(%rax,%rax)

0x10f4cae8d <+77>: nopl (%rax)

Not really sure how to debug this besides trying to build my own version of python and see if I can repro (I don’t have this issue on linux). I’ve also checked the sys.abiflags and both binaries have the same ones.

Has anyone experienced this before or has any pointers to debug it?

  • Afonso

Hey António,

If I understand correctly, you're trying to mix between two versions
of the Python interpreter and library. That's not something that's
supported and has always been an issue. Internally we get the
occasional bug report where somebody install python via homebrew or
python.org and the corresponding interpreter ends up first in their
path because /usr/local/bin comes before /usr/bin. The same is true if
you build your own LLDB, if you link against Homebrew's Python3 dylib,
you need to use the Homebrew interpreter. In CMake we try really hard
to ensure those two are in sync for when we run the (Python) test
suite.

On macOS Catalina, there's a shim for python3 in /usr/bin/ that will
launch the interpreter from Xcode, which matches what the LLDB from
Xcode is linked against.

Given that this is an issue that comes up frequently, I'm going to add
a bit of documentation about this on the LLDB website.

Best,
Jonas

I've put up a patch to extend the documentation: https://reviews.llvm.org/D70252

Please have a look let me know if you have any comments or suggestions!