SystemInitializerCommon::Initialize called twice with shared llvm build

Hi,

I am doing an llvm build with shared libraries and looks like due to
some weird interaction SystemInitializerCommon::Initialize() ends up
being called twice. This is how I produce:

λ lldb
(lldb) file ls
Current executable set to 'ls' (x86_64).
(lldb) r
<hangs>

Looking under with strace shows that lldb-server dies with:

[pid 15128] write(2, "lldb-server:
../lib/Support/ErrorHandling.cpp:50: void
llvm::install_fatal_error_handler(fatal_error_handler_t, void *):
Assertion `!ErrorHandler && \"Error handler already registered!\\n\"'
failed.\n", 196) = 196

I confirmed that indeed SystemInitializerCommon::Initialize() is being
called twice.

If I build llvm as static then the error goes away.

ismail

And backtrace looks like this:

#0 0x00007f80e416c638 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:55
#1 0x00007f80e416da1a in __GI_abort () at abort.c:78
#2 0x00007f80e41653a7 in __assert_fail_base (fmt=<optimized out>,
assertion=assertion@entry=0x7f80e4db072e "!ErrorHandler && \"Error
handler already registered!\\n\"", file=file@entry=0x7f80e4db0765
"../lib/Support/ErrorHandling.cpp", line=line@entry=50,
function=function@entry=0x7f80e4db0786 "void
llvm::install_fatal_error_handler(fatal_error_handler_t, void *)") at
assert.c:92
#3 0x00007f80e4165452 in __GI___assert_fail (assertion=0x7f80e4db072e
"!ErrorHandler && \"Error handler already registered!\\n\"",
file=0x7f80e4db0765 "../lib/Support/ErrorHandling.cpp", line=50,
function=0x7f80e4db0786 "void
llvm::install_fatal_error_handler(fatal_error_handler_t, void *)") at
assert.c:101
#4 0x00007f80e4d5466a in llvm::install_fatal_error_handler
(handler=<optimized out>, user_data=<optimized out>) at
../lib/Support/ErrorHandling.cpp:50
#5 0x00007f80e56d57c7 in
lldb_private::SystemInitializerCommon::Initialize (this=<optimized

) at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:101

#6 0x00007f80e54fb5cd in
lldb_private::SystemInitializerFull::Initialize (this=0x4751) at
../tools/lldb/source/API/SystemInitializerFull.cpp:226
#7 0x00007f80e56d5a3a in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x7f80e54c2bf0
<LoadPlugin(std::__1::shared_ptr<lldb_private::Debugger> const&,
lldb_private::FileSpec const&, lldb_private::Error&)>) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#8 0x00007f80e54c2bc7 in lldb::SBDebugger::Initialize () at
../tools/lldb/source/API/SBDebugger.cpp:123
#9 0x00007f80e55309d8 in _wrap_SBDebugger_Initialize (args=<optimized

) at tools/lldb/scripts/LLDBWrapPython.cpp:23596

#10 0x00007f80df5d7fe7 in call_function (oparg=<optimized out>,
pp_stack=0x7ffc2ad26550) at Python/ceval.c:4035
#11 PyEval_EvalFrameEx () at Python/ceval.c:2681
#12 0x00007f80df5da642 in fast_function (nk=<optimized out>,
na=<optimized out>, n=0, pp_stack=0x7ffc2ad26690, func=0x7f80d8e68938)
at Python/ceval.c:4121
#13 call_function (oparg=<optimized out>, pp_stack=0x7ffc2ad26690) at
Python/ceval.c:4056
#14 PyEval_EvalFrameEx () at Python/ceval.c:2681
#15 0x00007f80df5d6862 in PyEval_EvalCodeEx () at Python/ceval.c:3267
#16 0x00007f80df5fa2a6 in PyEval_EvalCode (co=co@entry=0x7f80d8dac7b0,
globals=globals@entry=0x7f80e5e71168,
locals=locals@entry=0x7f80e5e71168) at Python/ceval.c:669
#17 0x00007f80df600be8 in PyImport_ExecCodeModuleEx () at Python/import.c:713
#18 0x00007f80df6008f7 in load_source_module () at Python/import.c:1103
#19 0x00007f80df600d37 in load_package () at Python/import.c:1170
#20 0x00007f80df5dca8d in import_submodule (fullname=0x19f4ca0 "lldb",
subname=<optimized out>, mod=0x7f80df87bf40 <_Py_NoneStruct>) at
Python/import.c:2704
#21 load_next (p_buflen=<synthetic pointer>, buf=0x19f4ca0 "lldb",
p_name=<synthetic pointer>, altmod=<optimized out>, mod=<optimized

) at Python/import.c:2519

#22 import_module_level (locals=<optimized out>, level=<optimized

, fromlist=0x7f80df87bf40 <_Py_NoneStruct>, globals=<optimized
, name=<optimized out>) at Python/import.c:2228

#23 PyImport_ImportModuleLevel () at Python/import.c:2292
#24 0x00007f80df5d5bf8 in builtin___import__ () at Python/bltinmodule.c:49
#25 0x00007f80df5c236b in PyObject_Call () at Objects/abstract.c:2529
#26 0x00007f80df5da890 in PyEval_CallObjectWithKeywords () at
Python/ceval.c:3904
#27 0x00007f80df5d82f7 in PyEval_EvalFrameEx () at Python/ceval.c:2348
#28 0x00007f80df5d6862 in PyEval_EvalCodeEx () at Python/ceval.c:3267
#29 0x00007f80df5fa2a6 in PyEval_EvalCode (co=co@entry=0x7f80e5f4cdb0,
globals=globals@entry=0x7f80e5f07d70,
locals=locals@entry=0x7f80e5f07d70) at Python/ceval.c:669
#30 0x00007f80df60437d in run_mod (mod=<optimized out>,
filename=filename@entry=0x7f80df60eaeb "<string>",
globals=globals@entry=0x7f80e5f07d70,
locals=locals@entry=0x7f80e5f07d70, flags=flags@entry=0x0,
arena=arena@entry=0x19c8520) at Python/pythonrun.c:1371
#31 0x00007f80df604414 in PyRun_StringFlags
(str=str@entry=0x7f80e59e2d46 "sys.dont_write_bytecode = 1; import
lldb.embedded_interpreter; from lldb.embedded_interpreter import
run_python_interpreter; from lldb.embedded_interpreter import
run_one_line", start=start@entry=257, globals=0x7f80e5f07d70,
locals=0x7f80e5f07d70, flags=flags@entry=0x0) at
Python/pythonrun.c:1334
#32 0x00007f80df5a70ed in PyRun_SimpleStringFlags
(command=0x7f80e59e2d46 "sys.dont_write_bytecode = 1; import
lldb.embedded_interpreter; from lldb.embedded_interpreter import
run_python_interpreter; from lldb.embedded_interpreter import
run_one_line", flags=0x0) at Python/pythonrun.c:975
#33 0x00007f80e5705823 in
lldb_private::ScriptInterpreterPython::InitializePrivate () at
../tools/lldb/source/Interpreter/ScriptInterpreterPython.cpp:3140
#34 0x00007f80e56d587a in
lldb_private::SystemInitializerCommon::Initialize (this=<optimized

) at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:144

#35 0x00007f80e56d5a3a in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x0) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#36 0x00000000004160fe in initialize () at
../tools/lldb/tools/lldb-server/lldb-server.cpp:38
#37 0x0000000000415fde in main (argc=6, argv=0x7ffc2ad26f78) at
../tools/lldb/tools/lldb-server/lldb-server.cpp:62

And setting a break point on
lldb_private::SystemInitializerCommon::Initialize() shows that 2
initializations indeed:

First one:

#0 lldb_private::SystemInitializerCommon::Initialize (this=0x1ccec10)
at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:96
#1 0x00007f76d64385cd in
lldb_private::SystemInitializerFull::Initialize (this=0x1ccec10) at
../tools/lldb/source/API/SystemInitializerFull.cpp:226
#2 0x00007f76d6612a3a in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x7f76d63ffbf0
<LoadPlugin(std::__1::shared_ptr<lldb_private::Debugger> const&,
lldb_private::FileSpec const&, lldb_private::Error&)>) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#3 0x00007f76d63ffbc7 in lldb::SBDebugger::Initialize () at
../tools/lldb/source/API/SBDebugger.cpp:123
#4 0x00000000004063bb in main (argc=1, argv=0x7ffc50c8c948,
envp=0x7f76d6438f80 <LLDBSwigPythonWatchpointCallbackFunction(char
const*, char const*, lldb::StackFrameSP const&, lldb::WatchpointSP
const&)>) at ../tools/lldb/tools/driver/Driver.cpp:1223

Second one (this happens after I do "file ls" and "r") :

#0 lldb_private::SystemInitializerCommon::Initialize (this=0x1096850)
at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:96
#1 0x00007f11ba252a3a in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x0) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#2 0x00000000004160fe in initialize () at
../tools/lldb/tools/lldb-server/lldb-server.cpp:38
#3 0x0000000000415fde in main (argc=6, argv=0x7ffc6179b168) at
../tools/lldb/tools/lldb-server/lldb-server.cpp:62

Hi,

I have tried to follow your setup as closely as possible and I still
am not able to reproduce this. :confused:

Also, the backtrace you provided are quite strange also. The first one
terminates and lldb-server main, while the second one ends in lldb.
This is to be expected if they are from different processes since both
of them do initialization. Are you sure these backtraces are from the
same process? If so, then please find out why are both main()
functions being executed.

pl

Hi,

Problem seems to be lldb-server itself, looks like python code tries
to re-initialize, see :

λ gdb --args lldb-server g
(gdb) b lldb_private::SystemInitializerCommon::Initialize()
Function "lldb_private::SystemInitializerCommon::Initialize()" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (lldb_private::SystemInitializerCommon::Initialize()) pending.
(gdb) r
Starting program: /opt/clang/bin/lldb-server g
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

Breakpoint 1, lldb_private::SystemInitializerCommon::Initialize
(this=0x1f88670) at
../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:96
96 Log::Initialize();
(gdb) bt
#0 lldb_private::SystemInitializerCommon::Initialize (this=0x1f88670)
at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:96
#1 0x00007fc7709e56aa in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x0) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#2 0x000000000041707e in initialize () at
../tools/lldb/tools/lldb-server/lldb-server.cpp:38
#3 0x0000000000416f5e in main (argc=2, argv=0x7ffc22f92208) at
../tools/lldb/tools/lldb-server/lldb-server.cpp:62
(gdb) c
Continuing.
[New process 26104]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Thread 0x7fc77128c740 (LWP 26104) is executing new program: /bin/bash
Thread 0x7fc77128c740 (LWP 26104) is executing new program: /bin/bash
[New process 26107]
process 26107 is executing new program: /usr/bin/getopt
[Inferior 3 (process 26107) exited with code 04]
(gdb) [New process 26108]
process 26108 is executing new program: /usr/bin/getopt
[Inferior 4 (process 26108) exited normally]
[New process 26109]
process 26109 is executing new program: /usr/bin/basename
[Inferior 5 (process 26109) exited normally]
[New process 26110]
process 26110 is executing new program: /usr/bin/basename
[Inferior 6 (process 26110) exited normally]
[New process 26111]
process 26111 is executing new program: /usr/bin/basename
[Inferior 7 (process 26111) exited normally]
[New process 26112]
process 26112 is executing new program: /usr/bin/basename
[Inferior 8 (process 26112) exited normally]
[New process 26113]
process 26113 is executing new program: /usr/bin/basename
[Inferior 9 (process 26113) exited normally]
[New process 26114]
process 26114 is executing new program: /usr/bin/basename
[Inferior 10 (process 26114) exited normally]
[New process 26115]
process 26115 is executing new program: /usr/bin/basename
[Inferior 11 (process 26115) exited normally]
[New process 26116]
process 26116 is executing new program: /usr/bin/basename
[Inferior 12 (process 26116) exited normally]
[New process 26117]
[New process 26118]
[New process 26119]
[Inferior 14 (process 26118) exited normally]
process 26119 is executing new program: /usr/bin/sed
[Inferior 15 (process 26119) exited normally]
[Inferior 13 (process 26117) exited normally]
[Inferior 2 (process 26104) exited normally]
[New process 26120]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Thread 0x7fc77128c740 (LWP 26120) is executing new program: /bin/bash
[New process 26121]
process 26121 is executing new program: /sbin/ldconfig
[Inferior 17 (process 26121) exited normally]
[Inferior 16 (process 26120) exited normally]

Breakpoint 1, lldb_private::SystemInitializerCommon::Initialize
(this=0x1fb7ed0) at
../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:96
96 Log::Initialize();
bt
#0 lldb_private::SystemInitializerCommon::Initialize (this=0x1fb7ed0)
at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:96
#1 0x00007fc77080a4fd in
lldb_private::SystemInitializerFull::Initialize (this=0x1fb7ed0) at
../tools/lldb/source/API/SystemInitializerFull.cpp:226
#2 0x00007fc7709e56aa in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x7fc7707d1ac0
<LoadPlugin(std::__1::shared_ptr<lldb_private::Debugger> const&,
lldb_private::FileSpec const&, lldb_private::Error&)>) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#3 0x00007fc7707d1a97 in lldb::SBDebugger::Initialize () at
../tools/lldb/source/API/SBDebugger.cpp:123
#4 0x00007fc77083f908 in _wrap_SBDebugger_Initialize (args=<optimized

) at tools/lldb/scripts/LLDBWrapPython.cpp:23598

#5 0x00007fc76a8bffe7 in call_function (oparg=<optimized out>,
pp_stack=0x7ffc22f917e0) at Python/ceval.c:4035
#6 PyEval_EvalFrameEx () at Python/ceval.c:2681
#7 0x00007fc76a8c2642 in fast_function (nk=<optimized out>,
na=<optimized out>, n=0, pp_stack=0x7ffc22f91920, func=0x7fc763ae99b0)
at Python/ceval.c:4121
#8 call_function (oparg=<optimized out>, pp_stack=0x7ffc22f91920) at
Python/ceval.c:4056
#9 PyEval_EvalFrameEx () at Python/ceval.c:2681
#10 0x00007fc76a8be862 in PyEval_EvalCodeEx () at Python/ceval.c:3267
#11 0x00007fc76a8e22a6 in PyEval_EvalCode (co=co@entry=0x7fc7642517b0,
globals=globals@entry=0x7fc77117bd70,
locals=locals@entry=0x7fc77117bd70) at Python/ceval.c:669
#12 0x00007fc76a8e8be8 in PyImport_ExecCodeModuleEx () at Python/import.c:713
#13 0x00007fc76a8e88f7 in load_source_module () at Python/import.c:1103
#14 0x00007fc76a8e8d37 in load_package () at Python/import.c:1170
#15 0x00007fc76a8c4a8d in import_submodule (fullname=0x2016e50 "lldb",
subname=<optimized out>, mod=0x7fc76ab63f40 <_Py_NoneStruct>) at
Python/import.c:2704
#16 load_next (p_buflen=<synthetic pointer>, buf=0x2016e50 "lldb",
p_name=<synthetic pointer>, altmod=<optimized out>, mod=<optimized

) at Python/import.c:2519

#17 import_module_level (locals=<optimized out>, level=<optimized

, fromlist=0x7fc76ab63f40 <_Py_NoneStruct>, globals=<optimized
, name=<optimized out>) at Python/import.c:2228

#18 PyImport_ImportModuleLevel () at Python/import.c:2292
#19 0x00007fc76a8bdbf8 in builtin___import__ () at Python/bltinmodule.c:49
#20 0x00007fc76a8aa36b in PyObject_Call () at Objects/abstract.c:2529
#21 0x00007fc76a8c2890 in PyEval_CallObjectWithKeywords () at
Python/ceval.c:3904
#22 0x00007fc76a8c02f7 in PyEval_EvalFrameEx () at Python/ceval.c:2348
#23 0x00007fc76a8be862 in PyEval_EvalCodeEx () at Python/ceval.c:3267
#24 0x00007fc76a8e22a6 in PyEval_EvalCode (co=co@entry=0x7fc77116a3b0,
globals=globals@entry=0x7fc77121ad70,
locals=locals@entry=0x7fc77121ad70) at Python/ceval.c:669
#25 0x00007fc76a8ec37d in run_mod (mod=<optimized out>,
filename=filename@entry=0x7fc76a8f6aeb "<string>",
globals=globals@entry=0x7fc77121ad70,
locals=locals@entry=0x7fc77121ad70, flags=flags@entry=0x0,
arena=arena@entry=0x2051da0) at Python/pythonrun.c:1371
#26 0x00007fc76a8ec414 in PyRun_StringFlags
(str=str@entry=0x7fc770cf45b6 "sys.dont_write_bytecode = 1; import
lldb.embedded_interpreter; from lldb.embedded_interpreter import
run_python_interpreter; from lldb.embedded_interpreter import
run_one_line", start=start@entry=257, globals=0x7fc77121ad70,
locals=0x7fc77121ad70, flags=flags@entry=0x0) at
Python/pythonrun.c:1334
#27 0x00007fc76a88f0ed in PyRun_SimpleStringFlags
(command=0x7fc770cf45b6 "sys.dont_write_bytecode = 1; import
lldb.embedded_interpreter; from lldb.embedded_interpreter import
run_python_interpreter; from lldb.embedded_interpreter import
run_one_line", flags=0x0) at Python/pythonrun.c:975
#28 0x00007fc770a15ac3 in
lldb_private::ScriptInterpreterPython::InitializePrivate () at
../tools/lldb/source/Interpreter/ScriptInterpreterPython.cpp:3140
#29 0x00007fc7709e54ea in
lldb_private::SystemInitializerCommon::Initialize (this=<optimized

) at ../tools/lldb/source/Initialization/SystemInitializerCommon.cpp:144

#30 0x00007fc7709e56aa in
lldb_private::SystemLifetimeManager::Initialize (this=<optimized out>,
initializer=..., plugin_callback=0x0) at
../tools/lldb/source/Initialization/SystemLifetimeManager.cpp:43
#31 0x000000000041707e in initialize () at
../tools/lldb/tools/lldb-server/lldb-server.cpp:38
#32 0x0000000000416f5e in main (argc=2, argv=0x7ffc22f92208) at
../tools/lldb/tools/lldb-server/lldb-server.cpp:62

Hi Ismall,

lldb-server don’t use python at all so a possible fix would be to move the python initialization from SystemInitializerCommon to SystemInitializerFull. This change will most likely fix your issue and it is a reasonable change to make (also independently of this issue) but I am not sure if it addresses the root cause of the issue you are seeing. So please feel free to make the change but don’t be surprised if the issue appears in a different scenario also.

Tamas

Hi,