IOHandler branch has been merged into top of tree

We will need everyone to make sure their buildbots are ok and get any issues fixed. This is the one and only time the input/output and error streams will be changed in this drastic of a way. The benefits are many:

1 - Input/Output/Error streams are now handled as real streams not a push style input
2 - auto completion in python embedded interpreter
3 - multi-line input for "script" and "expression" commands now allow you to edit previous/next lines using up and down arrow keys and this makes multi-line input actually a viable thing to use
4 - it is now possible to use curses to drive LLDB (please try the "gui" command)

We will need to deal with and fix any buildbot failures and tests and arise now that input/output and error are correctly hooked up in all cases.

Greg Clayton

FYI - these are the tests that are failing for me on Ubuntu 12.04 x86_64 with top of tree after the iohandler merge sync:

Failing Tests (6)
FAIL: LLDB (suite) :: TestAliases.py (Linux tfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)
FAIL: LLDB (suite) :: TestCommandScript.py (Linux tfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)
FAIL: LLDB (suite) :: TestConvenienceVariables.py (Linux tfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)
FAIL: LLDB (suite) :: TestConditionalBreak.py (Linux tfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterCategories.py (Linux tfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)
FAIL: LLDB (suite) :: TestDataFormatterGlobals.py (Linux tfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)

I’ll start looking at those tomorrow if somebody doesn’t beat me to them.

Here’s what I’m finding on our setup:

FAIL: LLDB (suite) :: TestConvenienceVariables.py (Linuxtfiala2.mtv.corp.google.com 3.2.5-gg1336 #1 SMP Thu Aug 29 02:37:18 PDT 2013 x86_64 x86_64)

This test is failing most of the time. There appears to be some kind of race on the readline/libedit functionality. The embedded python that is being tested by the script is using the system readline library, which is different than the new libedit that we’re using in the lldb executable. The embedded python is core dumping (seg fault) around reading a string. I’m getting a back trace like so from the lldb test process that seg faulted:

#0 0x00007f8749507e87 in ?? () from /usr/lib/python2.7/lib-dynload/readline.so
#1 0x00007f874b6c3610 in PyOS_Readline () from /usr/lib/libpython2.7.so.1.0
#2 0x00007f874b640fad in ?? () from /usr/lib/libpython2.7.so.1.0
#3 0x00007f874b60c5d5 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#4 0x00007f874b5cc6b5 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#5 0x00007f874b60c650 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#6 0x00007f874b5cc6b5 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#7 0x00007f874b60c650 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#8 0x00007f874b5cc6b5 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#9 0x00007f874b60c650 in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#10 0x00007f874b60d37b in PyEval_EvalFrameEx () from /usr/lib/libpython2.7.so.1.0
#11 0x00007f874b5cc6b5 in PyEval_EvalCodeEx () from /usr/lib/libpython2.7.so.1.0
#12 0x00007f874b5cc9e2 in PyEval_EvalCode () from /usr/lib/libpython2.7.so.1.0
#13 0x00007f874b5cca7c in PyRun_StringFlags () from /usr/lib/libpython2.7.so.1.0
#14 0x00007f874b5cd6cb in PyRun_SimpleStringFlags () from /usr/lib/libpython2.7.so.1.0
#15 0x00007f874e71cbf0 in IOHandlerPythonInterpreter::Run (this=0x19e79c0)
at /mnt/ssd/work/svn/lgs/llvm/tools/lldb/source/Interpreter/ScriptInterpreterPython.cpp:747
#16 0x00007f874e4d36bc in lldb_private::Debugger::ExecuteIOHanders (this=0x17cc400)
at /mnt/ssd/work/svn/lgs/llvm/tools/lldb/source/Core/Debugger.cpp:865
#17 0x00007f874e647c16 in lldb_private::CommandInterpreter::RunCommandInterpreter (this=0x17e03f0,
auto_handle_events=true, spawn_thread=false)
at /mnt/ssd/work/svn/lgs/llvm/tools/lldb/source/Interpreter/CommandInterpreter.cpp:3002
#18 0x00007f874e3c93f5 in lldb::SBDebugger::RunCommandInterpreter (this=0x7fff9fb8c2a0,
auto_handle_events=true, spawn_thread=false)
at /mnt/ssd/work/svn/lgs/llvm/tools/lldb/source/API/SBDebugger.cpp:961
#19 0x000000000040f5ae in Driver::MainLoop (this=0x7fff9fb8c280)
at /mnt/ssd/work/svn/lgs/llvm/tools/lldb/tools/driver/Driver.cpp:946
#20 0x000000000040f8f4 in main (argc=3, argv=0x7fff9fb8c468, envp=0x7fff9fb8c488)
at /mnt/ssd/work/svn/lgs/llvm/tools/lldb/tools/driver/Driver.cpp:1039

On Ubuntu 12.04 LTS x86_64, the /usr/lib/python2.7/lib-dynload/readline.so links against GNU readline:

tfiala@tfiala2:/usr/lib/python2.7/lib-dynload$ ldd readline.so
linux-vdso.so.1 => (0x00007fff137f1000)
libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007fdd7d66c000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdd7d44f000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdd7d08e000)
libgcc_s.so.1 => /usr/local/gcc/gcc-current/lib64/libgcc_s.so.1 (0x00007fdd7ce78000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fdd7cc51000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdd7dad6000)

That makes me think that the embedded python interpreter is likely fighting with libedit.so in some way. My lldb is built with the latest config-enabled libedit, as we found the Ubuntu 12.04 LTS libedit-dev package was too old for some of the newer iohandler features:

tfiala@tfiala2:~/lldb/svn/lgs/build/Debug+Asserts/bin$ ldd lldb
linux-vdso.so.1 => (0x00007fff8c579000)
libedit.so.0 => /usr/local/google/home/tfiala/lldb/tools/libedit/linux-x86_64/lib/libedit.so.0 (0x00007f00fdac3000)
liblldb.so => /mnt/ssd/work/svn/lgs/build/Debug+Asserts/bin/./…/lib/liblldb.so (0x00007f00f87c3000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f00f858d000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f00f8370000)
libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f00f8148000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f00f7f40000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f00f7d3c000)
libstdc++.so.6 => /usr/local/gcc/gcc-current/lib64/libstdc++.so.6 (0x00007f00f7a38000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f00f773c000)
libgcc_s.so.1 => /usr/local/gcc/gcc-current/lib64/libgcc_s.so.1 (0x00007f00f7526000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f00f7165000)
libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f00f6f62000)
libpython2.7.so.1.0 => /usr/lib/libpython2.7.so.1.0 (0x00007f00f6a65000)
libncurses.so.5 => /lib/x86_64-linux-gnu/libncurses.so.5 (0x00007f00f6843000)
libpanel.so.5 => /usr/lib/x86_64-linux-gnu/libpanel.so.5 (0x00007f00f663f000)
/lib64/ld-linux-x86-64.so.2 (0x00007f00fdcf6000)
libssl.so.1.0.0 => /lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f00f63e0000)
libcrypto.so.1.0.0 => /lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f00f6005000)

I’m not yet sure if this is an inherent issue with two pieces of low level code (libedit & readline) both thinking they have sole access to the input/output file descriptors (which may require them both using readline or libedit, but not both), or just a bug in assumptions on our new iohandler input handling. I’m going to see if I can get lldb to build with readline first and see if that changes the behavior. It may convert a seg fault into a failing test, which may be easier to diagnose, or it might outright solve the problem. Back after I check out the readline route.

Building with libreadline-dev on Ubuntu 12.04 LTS x86_64 isn’t going to work. We’re using editline-specific functionality.

I’m going to try the other approach now, which is building and using a python that is built against libedit.

Digging through the python readline.c code, the libedit support appears to be hardcoded to working with APPLE #ifdefs — I am not seeing an easy way to build python with libedit support under Linux.

Any ideas here? It’s possible our issue is not so much a libedit/libreadline issue and more just faulty assumptions (possibly in Python’s readline.so) about atomic operations that are causing the crash. I was just trying to get both Python and LLDB to be using the same one.

-Todd

I was able to address the TestConvenienceVariables.py break with this patch (thanks for some clues on that, Greg!):

Index: test/dotest.py

I was able to address the TestConvenienceVariables.py break with this patch (thanks for some clues on that, Greg!):

Index: test/dotest.py

--- test/dotest.py (revision 200343)
+++ test/dotest.py (working copy)
@@ -873,10 +873,13 @@
     pluginPath = os.path.join(scriptPath, 'plugins')
     pexpectPath = os.path.join(scriptPath, 'pexpect-2.4')

- # Append script dir, plugin dir, and pexpect dir to the sys.path.
+ # Put embedded pexpect at front of the load path so we ensure we
+ # use that version.
+ sys.path.insert(0, pexpectPath)
+
+ # Append script dir and plugin dir to the sys.path.
     sys.path.append(scriptPath)
     sys.path.append(pluginPath)
- sys.path.append(pexpectPath)

     # This is our base name component.
     base = os.path.abspath(os.path.join(scriptPath, os.pardir))
Index: source/Interpreter/embedded_interpreter.py

--- source/Interpreter/embedded_interpreter.py (revision 200343)
+++ source/Interpreter/embedded_interpreter.py (working copy)
@@ -86,7 +86,7 @@
                 code.interact(banner="Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.", readfunc=readfunc_stdio, local=local_dict)
         else:
             # We have a real interactive terminal
- code.interact(banner="Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.", local=local_dict)
+ code.interact(banner="Python Interactive Interpreter. To exit, type 'quit()', 'exit()' or Ctrl-D.", readfunc=readfunc_stdio, local=local_dict)
     except SystemExit as e:
         global g_builtin_override_called
         if not g_builtin_override_called:

So we can't checkin the code above where we set "readfunc=readfunc_stdio" as this disables all auto completion as it won't use readline + rlcompleter if this is enabled. This simply identifies that the problem has something to do with readline...

So we still need to figure out what is going wrong on linux with readline. You could import "platform" in python and and set "readfunc=readfunc_stdio" for now for non darwin platforms (or anything that is readline based and causing the problem) just to get things working, but I am guessing people will really want the auto completion in the embedded python interpreter...

Yeah I figured that wouldn’t be the final solution. But I’m guessing this is exposing some kind of bad interaction that is going to (potentially) happen anywhere that python links with libreadline, which is everywhere that APPLE isn’t defined (so maybe wider than Linux).

So we still need to figure out what is going wrong on linux with readline.

Right. At this point it might even be a manifestation of how test input/output is interacting with it. I haven’t caught it happening outside of the test environment yet (although I’ve only tried a handful of times), although nothing in the nature of the crash screams test-only since this is crashing in a python module using readline.

You could import “platform” in python and and set “readfunc=readfunc_stdio” for now for non darwin platforms (or anything that is readline based and causing the problem) just to get things working, but I am guessing people will really want the auto completion in the embedded python interpreter…

Ok - I may also try to work it in so that it only disables auto completion in the test environment. If we started getting bug reports around it outside of the test environment, we would know this is a bigger issue.

-Todd

I am getting these error when building with cmake. Autoconf seems ok. Any idea what can be the problem?

../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for llvm::ModulePass'
../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for clang::ExternalASTSource'
../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for llvm::JITMemoryManager'
../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for clang::ASTImporter'
../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for clang::DiagnosticConsumer'
../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for llvm::MemoryObject'
../../../../lib/liblldb.so.3.5: undefined reference to `typeinfo for clang::SemaConsumer'

Regards,
Abid

RTTI is disabled in all the llvm/clang/lldb builds. When something is looking for typeinfo, it tends to mean some file was compiled thinking RTTI was enabled.

The bug that I just fixed with ScriptInterpreterPython reference counting of stdin/stderr/stdout appears to have fixed the libedit/libreadline issues we were initially investigating :slight_smile:

That now maybe makes more sense in light of the fix since it may have been trying to do something with a file (maybe via a weak reference) that really should have been dead at that point.

At this point I have no more test failures on top of tree lldb at r200486.

We are currently also having the same problem when building lldb for linux.
The work around we are using is to build clang, llvm and lldb with "-fno-rtti" as part of CMAKE_CXX_FLAGS, which will bypass this problem.
My guess is that something has been added to one of these project which doesn't build with -fno-rtti, or it has been removed somewhere.

It seems that problem was introduced by the following commit on 28th.
3303370cd028ecb3c57820e2e994f3d3099c0607
[CMake] Enhance llvm_update_compile_flags(name sources) to handle LLVM_REQUIRES_EH and LLVM_REQUIRES_RTTI.

I will check with the author.

Regards,
Abid