dyld: lazy symbol binding failed: fast lazy bind offset out of range

Jack_Howarth1 · October 21, 2012, 4:27pm

Nick,
While enhancing the fink llvm32 packaging to support the polly tool, I ran into
a dyld failure when the dragonegg plugin tries to load LLVMPolly.so plugin...

dyld: lazy symbol binding failed: fast lazy bind offset out of range (39257, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
dyld: fast lazy bind offset out of range (39257, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin11.4.2/4.7.2/cc1
Trace/BPT trap

which backtraces as...

(gdb) bt
#0 0x00007fff5fc0106d in __dyld_dyld_fatal_error ()
#1 0x00007fff5fc048c8 in __dyld__ZN4dyld4haltEPKc ()
#2 0x00007fff5fc04948 in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
#3 0x00007fff90af7716 in dyld_stub_binder_ ()
#4 0x0000000100f3b2c0 in Json::Value::maxUInt ()
#5 0x00007fff5fc0fda6 in __dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE ()
#6 0x00007fff5fc0faf2 in __dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
#7 0x00007fff5fc0d2e4 in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEjRNS_21InitializerTimingListE ()
#8 0x00007fff5fc0e0b7 in __dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextERNS_21InitializerTimingListE ()
#9 0x00007fff5fc031b9 in __dyld__ZN4dyld15runInitializersEP11ImageLoader ()
#10 0x00007fff5fc09657 in __dyld_dlopen ()
#11 0x00007fff90af795b in dlopen ()
#12 0x0000000142f3888e in llvm::sys::DynamicLibrary::getPermanentLibrary ()
#13 0x0000000142f256bd in llvm::PluginLoader::operator= ()
#14 0x000000014280fbda in llvm::opt<llvm::PluginLoader, false, llvm::parser<std::string> >::handleOccurrence ()
#15 0x0000000142f1730c in CommaSeparateAndAddOccurence ()
#16 0x0000000142f1395f in ProvideOption ()
#17 0x0000000142f1224b in llvm::ParseCommandLineOptions ()
#18 0x000000014280d3ff in InitializeBackend ()
#19 0x000000014280c191 in llvm_emit_globals ()

This will be difficult to testcase since the FSF gcc 4.7.2 compiler is involved but I was hoping that you might be able
to make some guesses about the origin of the problem from the current polly and dragonegg source code. There doesn't
seem to be any reports of this error type (but then again few plugins, dragonegg, indirectly call additional plugins,
LLVMPolly. Thanks in advance for any hints.
Jack

Nick_Kledzik · October 22, 2012, 1:00am

Jack,

Some binary has an initializer which dyld is calling. Somehow the initializer gets to:
#4 0x0000000100f3b2c0 in Json::Value::maxUInt ()
which is calling a function in another dylib for the first time. When you call a function in another dylib, you actually jump through a (lazy) pointer. The pointer initially points to a helper which loads an index parameter specifying which function to bind, then jumps into dyld. Dyld is erroring because the index is out of range.

Either:
1) The binary is corrupt. You can check that by running:
dyldinfo -lazy_bind to see the lazy pointers
or
2) Something else is wrong with the initializer and it happens to be jumping to the helper (that jumps into dyld), but the index parameter is garbage. You'll need to step though this in the debugger.

-Nick

Jack_Howarth1 · October 22, 2012, 1:48pm

Jack,

Some binary has an initializer which dyld is calling. Somehow the initializer gets to:
#4 0x0000000100f3b2c0 in Json::Value::maxUInt ()
which is calling a function in another dylib for the first time. When you call a function in another dylib, you actually jump through a (lazy) pointer. The pointer initially points to a helper which loads an index parameter specifying which function to bind, then jumps into dyld. Dyld is erroring because the index is out of range.

Either:
1) The binary is corrupt. You can check that by running:
dyldinfo -lazy_bind to see the lazy pointers

Nick,
I attached dragonegg_so_lazy_bind.txt and LLVMPolly_so_lazy_bind.txt to http://llvm.org/bugs/show_bug.cgi?id=14140.
Is there anything significant in those lists of lazy pointers relative to the failure we are seeing? FYI, I see the same
failure for both x86_64-apple-darwin11 and x86_64-apple-darwin12 with current llvm/polly/dragonegg svn using Xcode 4.5.1's
linker.

or
2) Something else is wrong with the initializer and it happens to be jumping to the helper (that jumps into dyld), but the index parameter is garbage. You'll need to step though this in the debugger.

From the backtrace quoted below, can you suggest a logical place to set the breakpoint as the starting point to step through this code?

Thanks in advance.
Jack

Jack_Howarth1 · October 22, 2012, 3:07pm

> Jack,
>
> Some binary has an initializer which dyld is calling. Somehow the initializer gets to:
> #4 0x0000000100f3b2c0 in Json::Value::maxUInt ()
> which is calling a function in another dylib for the first time. When you call a function in another dylib, you actually jump through a (lazy) pointer. The pointer initially points to a helper which loads an index parameter specifying which function to bind, then jumps into dyld. Dyld is erroring because the index is out of range.
>
> Either:
> 1) The binary is corrupt. You can check that by running:
> dyldinfo -lazy_bind to see the lazy pointers

Nick,
I attached dragonegg_so_lazy_bind.txt and LLVMPolly_so_lazy_bind.txt to http://llvm.org/bugs/show_bug.cgi?id=14140.
Is there anything significant in those lists of lazy pointers relative to the failure we are seeing? FYI, I see the same
failure for both x86_64-apple-darwin11 and x86_64-apple-darwin12 with current llvm/polly/dragonegg svn using Xcode 4.5.1's
linker.

> or
> 2) Something else is wrong with the initializer and it happens to be jumping to the helper (that jumps into dyld), but the index parameter is garbage. You'll need to step though this in the debugger.

>From the backtrace quoted below, can you suggest a logical place to set the breakpoint as the starting point to step through this code?
Thanks in advance.
Jack

Nick,
I have attached a gdb walk log (LLVMPolly_load_walk.log) to http://llvm.org/bugs/show_bug.cgi?id=14140 which
steps from llvm-3.2/lib/Support/CommandLine.cpp, line 259. The tail end of the log is as follows...

(gdb)
llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x144002188 "/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe700) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
77 void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
(gdb)
Reading symbols for shared libraries ... done
dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

Let me know if there is something else I can provide to help debug this nested plugin loading issue on darwin.
Jack
ps This is for an llvm/polly debug cmake build with assertions on as well as dragonegg built with -g.

Nick_Kledzik · October 22, 2012, 5:17pm

Jack,

I looks like the code is calling dlopen() on LLVMPolly.so and it or something it links against has an initializer. The initialer is run before dlopen() returns and the crash is in the initializer. The message:

dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

means the initializer called something which made its way into the cc1 binary, probably it called some weak symbol which resolved in the main executable cc1.

In gdb you can set:
(gdb) set env DYLD_PRINT_INITIALIZERS
and dyld will print out the address of each initializer before calling it. You can then run again and set a break point at that address and step through it.

Also, the lazy bind info for LLVMPolly.so was a little odd in that all symbols were marked "flat-namespace". On MacOSX we normally record in which dylib each symbol should be found. The use of flat-namespace tells dyld to look everywhere for those symbols. This can cause the wrong symbol to be used at runtime. Was LLVMPolly.so build with -flat_namespace or -undefined dynamic_lookup?

-Nick

Jack_Howarth1 · October 22, 2012, 5:51pm

Jack,

I looks like the code is calling dlopen() on LLVMPolly.so and it or something it links against has an initializer. The initialer is run before dlopen() returns and the crash is in the initializer. The message:

dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

means the initializer called something which made its way into the cc1 binary, probably it called some weak symbol which resolved in the main executable cc1.

In gdb you can set:
(gdb) set env DYLD_PRINT_INITIALIZERS
and dyld will print out the address of each initializer before calling it. You can then run again and set a break point at that address and step through it.

Nick,
Without stepping, I get the following...

dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff93ec7d70 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec8190 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec9020 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ecbf00 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ece3b0 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93eebb40 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93f02ba0 in /usr/lib/libstdc++.6.dylib
dyld: calling -init function 0x0x7fff91c19830 in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff93ec7d70 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec8190 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec9020 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ecbf00 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ece3b0 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93eebb40 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93f02ba0 in /usr/lib/libstdc++.6.dylib
dyld: calling termination function 0x1000176ac in /System/Library/Perl/5.12/darwin-thread-multi-2level/CORE/libperl.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff93ec7d70 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec8190 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec9020 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ecbf00 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ece3b0 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93eebb40 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93f02ba0 in /usr/lib/libstdc++.6.dylib
dyld: calling -init function 0x0x7fff91c19830 in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
Reading symbols for shared libraries +++++++++++++................................. done
dyld: calling initializer function 0x7fff928cca81 in /usr/lib/libSystem.B.dylib
dyld: calling initializer function 0x7fff89899720 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff898bc3b0 in /usr/lib/libc++.1.dylib
dyld: calling initializer function 0x7fff93ec7d70 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec8190 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ec9020 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ecbf00 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93ece3b0 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93eebb40 in /usr/lib/libstdc++.6.dylib
dyld: calling initializer function 0x7fff93f02ba0 in /usr/lib/libstdc++.6.dylib
dyld: calling -init function 0x0x7fff91c19830 in /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
dyld: calling initializer function 0x1417d18b0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d1cb0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d1da0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d1ee0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d2090 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d21c0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d23b0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d2550 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d25e0 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x1417d2670 in /sw/lib/gmp5/libgmpxx.4.dylib
dyld: calling initializer function 0x14167d8b0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14167e750 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14167f820 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416811e0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141682ee0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141684700 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141686a30 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141688cd0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141689a50 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14168b500 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14168d2c0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14168d4c0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14168d600 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14168f570 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141695060 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141696f00 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416973a0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416a3650 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b3460 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b59a0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b5b50 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b5dd0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b6590 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b67d0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b69b0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416b6b00 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416c0420 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416d7da0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416d9a80 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416e37e0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416e3ef0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416e4230 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416e80a0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416f7260 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416f8eb0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416fa3d0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416fa810 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416fad90 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416fe8f0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1416ff460 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141700240 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141702950 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141703710 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141704000 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1417040e0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1417045e0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141706290 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141706910 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141706f00 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141708cb0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14170a6c0 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x14170aa90 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x141711a80 in /sw/lib/libppl.9.dylib
dyld: calling initializer function 0x1417cb790 in /sw/lib/libpwl.5.dylib
dyld: calling initializer function 0x1417cb8d0 in /sw/lib/libpwl.5.dylib
dyld: calling initializer function 0x1412853c0 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x1412b9300 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x1412da8a0 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x141314c90 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x14134ce00 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x141383d70 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x1413c0d90 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x1413fca10 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x14140e910 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x141425640 in /sw/lib/libppl_c.4.dylib
dyld: calling initializer function 0x14143b0a0 in /sw/lib/libppl_c.4.dylib
Reading symbols for shared libraries warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Aliasing.o" - no debug information available for "Aliasing.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Backend.o" - no debug information available for "Backend.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Cache.o" - no debug information available for "Cache.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/ConstantConversion.o" - no debug information available for "ConstantConversion.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Convert.o" - no debug information available for "Convert.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Debug.o" - no debug information available for "Debug.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/DefaultABI.o" - no debug information available for "DefaultABI.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Trees.o" - no debug information available for "Trees.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/TypeConversion.o" - no debug information available for "TypeConversion.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/bits_and_bobs.o" - no debug information available for "bits_and_bobs.cpp".

warning: Could not find object file "/sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/Target.o" - no debug information available for "Target.cpp".

. done
dyld: calling initializer function 0x146000ea0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14600cdf0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146029180 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146061960 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14609d590 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1460b0590 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1461d3da0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1461de2b0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1461e7a10 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146236cd0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1462aea90 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1462d9f10 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1462e8460 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1462f1430 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1463726e0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14639f920 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1463b3e40 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1463d51d0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146416d70 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146451750 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14646f4d0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146480270 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464989b0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14649d340 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464bdce0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464c7dc0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464cb230 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464d1430 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464dd8f0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1464ed3a0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146567500 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14659a960 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465a3bd0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465c2670 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465c84b0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465cf800 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465d4cf0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465e1b70 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465e8770 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465f3ec0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1465fd600 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146607aa0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146613ed0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14661bad0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146631790 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14663fb50 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146658f20 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1466636e0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146679ec0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146681ab0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146686e40 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146698320 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146699c60 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14669c550 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1466a7950 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1466b4c90 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1466ce7e0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467194b0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14671d510 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467213b0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146728430 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14672fb40 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146743280 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146755fe0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14675e0f0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146786f70 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14679d190 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467a4b30 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467aee60 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467dc0a0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467ea7d0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1467f0950 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146812f80 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146867f30 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x14688dea0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146929750 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146945600 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146948bd0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146978bb0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146998940 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1469ae280 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1469b0840 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x1469d8350 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146a49680 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146a748f0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146a78ab0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146a7f1c0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146a8e0f0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146aae960 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146ab17a0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146ab49e0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146abb910 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146ad1330 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146afa880 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146b2b5e0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146b3dbc0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146b99320 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146c04860 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146c18e50 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146cfb1e0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d5a3c0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d6b240 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d70920 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d76720 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d7a3a0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d800c0 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d88520 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146d98580 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146daee70 in /sw/lib/gcc4.7/lib/dragonegg.so
dyld: calling initializer function 0x146db34f0 in /sw/lib/gcc4.7/lib/dragonegg.so
GNU C (GCC) version 4.7.2 (x86_64-apple-darwin12.2.0)
compiled by GNU C version 4.7.2, GMP version 5.0.5, MPFR version 3.1.1, MPC version 1.0.1
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Versions of loaded plugins:
dragonegg: 3.2svn
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/sw/lib/gcc4.7/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/../../../../x86_64-apple-darwin12.2.0/include"
#include "..." search starts here:
#include <...> search starts here:
/sw/lib/gcc4.7/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/include
/sw/lib/gcc4.7/include
/sw/lib/gcc4.7/lib/gcc/x86_64-apple-darwin12.2.0/4.7.2/include-fixed
/usr/include
/System/Library/Frameworks
/Library/Frameworks
End of search list.
GNU C (GCC) version 4.7.2 (x86_64-apple-darwin12.2.0)
compiled by GNU C version 4.7.2, GMP version 5.0.5, MPFR version 3.1.1, MPC version 1.0.1
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Versions of loaded plugins:
dragonegg: 3.2svn
Compiler executable checksum: b3b092ad36f22724bf4c76dcddb14d3d
himenoBMTxpa.c: In function 'main':
himenoBMTxpa.c:79:5: warning: incompatible implicit declaration of built-in function 'strcpy' [enabled by default]
himenoBMTxpa.c: In function 'set_param':
himenoBMTxpa.c:226:5: warning: incompatible implicit declaration of built-in function 'exit' [enabled by default]
himenoBMTxpa.c: In function 'newMat':
himenoBMTxpa.c:239:5: warning: incompatible implicit declaration of built-in function 'malloc' [enabled by default]
himenoBMTxpa.c: In function 'clearMat':
himenoBMTxpa.c:248:5: warning: incompatible implicit declaration of built-in function 'free' [enabled by default]
Reading symbols for shared libraries ... done
dyld: calling initializer function 0x100eea5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007fff5fc0109d in __dyld_dyld_fatal_error ()

I'll upload a full walk from llvm-3.2/lib/Support/CommandLine.cpp, line 259 to http://llvm.org/bugs/show_bug.cgi?id=14140
in a bit.
Jack
ps In case it helps...

% otool -L /sw/opt/llvm-3.2/lib/LLVMPolly.so
/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

otool -L /sw/lib/gcc4.7/lib/dragonegg.so
/sw/lib/gcc4.7/lib/dragonegg.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

Jack_Howarth1 · October 22, 2012, 6:34pm

Jack,

I looks like the code is calling dlopen() on LLVMPolly.so and it or something it links against has an initializer. The initialer is run before dlopen() returns and the crash is in the initializer. The message:

dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

means the initializer called something which made its way into the cc1 binary, probably it called some weak symbol which resolved in the main executable cc1.

In gdb you can set:
(gdb) set env DYLD_PRINT_INITIALIZERS
and dyld will print out the address of each initializer before calling it. You can then run again and set a break point at that address and step through it.

Also, the lazy bind info for LLVMPolly.so was a little odd in that all symbols were marked "flat-namespace". On MacOSX we normally record in which dylib each symbol should be found. The use of flat-namespace tells dyld to look everywhere for those symbols. This can cause the wrong symbol to be used at runtime. Was LLVMPolly.so build with -flat_namespace or -undefined dynamic_lookup?

-Nick

Nick,
I have uploaded the full walk with 'set env DYLD_PRINT_INITIALIZERS'. It didn't seem very informative
as the dyld error occurs right after...

(gdb)
llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x142903da8 "/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6e0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
77 void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
(gdb)
Reading symbols for shared libraries ... done
dyld: calling initializer function 0x100eea5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007fff5fc0109d in __dyld_dyld_fatal_error ()

The linkage is...

% otool -L /sw/opt/llvm-3.2/lib/LLVMPolly.so
/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

and done with...

/sw/var/lib/fink/path-prefix-clang/c++ -fPIC -fvisibility-inlines-hidden -fno-exceptions -fno-rtti -g -bundle -Wl,-headerpad_max_install_names -Wl,-flat_namespace -Wl,-undefined -Wl,suppress -L/sw/lib -o ../../../lib/LLVMPolly.so CMakeFiles/LLVMPolly.dir/CodePreparation.cpp.o CMakeFiles/LLVMPolly.dir/DeadCodeElimination.cpp.o CMakeFiles/LLVMPolly.dir/IndependentBlocks.cpp.o CMakeFiles/LLVMPolly.dir/IndVarSimplify.cpp.o CMakeFiles/LLVMPolly.dir/MayAliasSet.cpp.o CMakeFiles/LLVMPolly.dir/Pocc.cpp.o CMakeFiles/LLVMPolly.dir/RegionSimplify.cpp.o CMakeFiles/LLVMPolly.dir/RegisterPasses.cpp.o CMakeFiles/LLVMPolly.dir/ScheduleOptimizer.cpp.o ../../../lib/libPollyAnalysis.a ../../../lib/libPollyCodeGen.a ../../../lib/libPollyExchange.a ../../../lib/libPollySupport.a ../../../lib/libPollyJSON.a /sw/lib/libisl.dylib /sw/lib/libgmp.dylib /sw/lib/libcloog-isl.dylib -lpthread

whereas the dragonegg.so is...

% otool -L /sw/lib/gcc4.7/lib/dragonegg.so
/sw/lib/gcc4.7/lib/dragonegg.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
/usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

using

clang++ -o dragonegg.so -bundle -undefined dynamic_lookup -Wall -Wextra -DENABLE_LLVM_PLUGINS -I/sw/opt/llvm-3.2/include -fPIC -fvisibility-inlines-hidden -g -Wall -W -Wno-unused-parameter -Wwrite-strings -pedantic -Wno-long-long -Wcovered-switch-default -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS \
  Aliasing.o Backend.o Cache.o ConstantConversion.o Convert.o Debug.o DefaultABI.o Trees.o TypeConversion.o bits_and_bobs.o Target.o \
  -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86CodeGen -lLLVMSelectionDAG -lLLVMAsmPrinter -lLLVMMCParser -lLLVMCodeGen -lLLVMX86Desc -lLLVMX86Info -lLLVMX86AsmPrinter -lLLVMX86Utils -lLLVMInstrumentation -lLLVMAsmParser -lLLVMBitWriter -lLLVMBitReader -lLLVMipo -lLLVMVectorize -lLLVMScalarOpts -lLLVMInstCombine -lLLVMTransformUtils -lLLVMipa -lLLVMAnalysis -lLLVMTarget -lLLVMMC -lLLVMObject -lLLVMCore -lLLVMSupport \
  -L/sw/opt/llvm-3.2/lib -L/sw/lib -lpthread -L/sw/lib

and cc1 is...

% otool -L /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1:
  /sw/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
  /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libcloog-isl.1.dylib (compatibility version 2.0.0, current version 2.2.0)
  /sw/lib/libisl.6.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libppl_c.4.dylib (compatibility version 5.0.0, current version 5.0.0)
  /sw/lib/libppl.9.dylib (compatibility version 10.0.0, current version 10.0.0)
  /sw/lib/libpwl.5.dylib (compatibility version 6.0.0, current version 6.0.0)
  /sw/lib/gmp5/libgmpxx.4.dylib (compatibility version 7.0.0, current version 7.5.0)
  /sw/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.1.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

I tried relinking /sw/opt/llvm-3.2/lib/LLVMPolly.so to have more of the linkages from
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

# otool -L /sw/opt/llvm-3.2/lib/LLVMPolly.so
/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
  /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libcloog-isl.1.dylib (compatibility version 2.0.0, current version 2.2.0)
  /sw/lib/libisl.6.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/gmp5/libgmpxx.4.dylib (compatibility version 7.0.0, current version 7.5.0)
  /sw/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.1.0)
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

and this didn't eliminate the error...

dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

Are there any tricks for identifying the order that the symbols are being resolved?
Jack

Nick_Kledzik · October 22, 2012, 6:40pm

Nick,
I have uploaded the full walk with 'set env DYLD_PRINT_INITIALIZERS'. It didn't seem very informative
as the dyld error occurs right after...

(gdb)
llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x142903da8 "/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6e0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
77 void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
(gdb)
Reading symbols for shared libraries ... done
dyld: calling initializer function 0x100eea5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so

You need to set a break point at 0x100eea5b0 and run again until that break is hit, then start stepping.

Are there any tricks for identifying the order that the symbols are being resolved?

(gdb) set env DYLD_PRINT_BINDINGS 1
will show each use of a symbol as it is bound. This can be *a lot* of information.

-Nick

Jack_Howarth1 · October 22, 2012, 6:52pm

Nick,
I will try one other thing first. I'll rebuild gcc47 without graphite support so cc1 has no linkages on cloog/ppl/isl
and the test it against a stock build of llvm32 to see if this suppresses the error.
Jack

Jack_Howarth1 · October 22, 2012, 8:01pm

Nick,
We can eliminate the linkages of ppl/isl/cloog in cc1 as the cause of the problem. I rebuilt the gcc47 package with...

Configured with: ../gcc-4.7.2/configure --prefix=/sw --prefix=/sw/lib/gcc4.7 --mandir=/sw/share/man --infodir=/sw/lib/gcc4.7/info --enable-languages=c,c++,fortran,lto,objc,obj-c++,java --with-gmp=/sw --with-libiconv-prefix=/sw --without-ppl --without-cloog --with-mpc=/sw --with-system-zlib --x-includes=/usr/X11R6/include --x-libraries=/usr/X11R6/lib --program-suffix=-fsf-4.7

to disable the graphite support such that cc1 is linked as...

% otool -L /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1:
  /sw/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
  /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.1.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Using a dragonegg.so plugin built against this compiler, still produces the error...

dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

when loading the /sw/opt/llvm-3.2/lib/LLVMPolly.so plugin. This is as much as I can reduce
the cc1 linkages so I'll have to manually hunt for the symbol now. Sigh.
Jack

Jack_Howarth1 · October 22, 2012, 11:34pm

Nick,
If I set 'break *0x100eea5b0', I didn't get any output from DYLD_PRINT_BINDINGS being set.
However if I set the breakpoint at PluginLoader.cpp:29 which is near the loading of LLVMPolly.so,
I get...

....
dyld: weak bind: LLVMPolly.so:0x100FB3130 = libc++.1.dylib:__ZdlPv, *0x100FB3130 = 0x7FFF898BD1AB
dyld: weak bind: LLVMPolly.so:0x100FB3138 = libc++.1.dylib:__Znam, *0x100FB3138 = 0x7FFF898BD173
dyld: weak bind: LLVMPolly.so:0x100FB3140 = libc++.1.dylib:__Znwm, *0x100FB3140 = 0x7FFF898BD0DF
dyld: calling initializer function 0x100ebb5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/c

The complete log is uploaded to http://llvm.org/bugs/attachment.cgi?id=9397 of http://llvm.org/bugs/show_bug.cgi?id=14140.
I would also note the following...

1) cc1 doesn't appear to be linked against any libstdc++.

/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1:
  /sw/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
  /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.1.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

whereas libPolly.so is always linked against the system libstdc++ shared library...

/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

2) The problem seems to be immune to which libstdc++ shared library that /sw/lib/gcc4.7/lib/dragonegg.so
is linked against. I have built dragonegg both with the FSF gcc-fsf-4.7 compiler to link in
the FSF libstdc++ as well with clang to use the system libstdc++. Both modes of building LLVMPolly.so
produce the same fast lazy binding failure.

Any suggestions on where to go from here in debugging this?
Jack

Duncan_Sands · October 23, 2012, 7:41am

Hi,

I looks like the code is calling dlopen() on LLVMPolly.so and it or something it links against has an initializer. The initialer is run before dlopen() returns and the crash is in the initializer. The message:

dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

means the initializer called something which made its way into the cc1 binary, probably it called some weak symbol which resolved in the main executable cc1.

this reminds me of something that came up recently (PR13981): both polly and
gcc are dynamically linked against libcloog (and usually against different
versions). When dlopen loads polly it resolves the libcloog symbols using
the version of libcloog that was loaded when gcc (the main binary) was started
up, causing all kinds of mayhem on linux. Maybe this could cause the kind of
issue you are seeing, especially if the two libcloog library versions differ?

Ciao, Duncan.

Jack_Howarth1 · October 23, 2012, 1:52pm

Hi,

I looks like the code is calling dlopen() on LLVMPolly.so and it or something it links against has an initializer. The initialer is run before dlopen() returns and the crash is in the initializer. The message:

dyld: fast lazy bind offset out of range (53437, max=7640) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

means the initializer called something which made its way into the cc1 binary, probably it called some weak symbol which resolved in the main executable cc1.

this reminds me of something that came up recently (PR13981): both polly and
gcc are dynamically linked against libcloog (and usually against different
versions). When dlopen loads polly it resolves the libcloog symbols using
the version of libcloog that was loaded when gcc (the main binary) was started
up, causing all kinds of mayhem on linux. Maybe this could cause the kind of
issue you are seeing, especially if the two libcloog library versions differ?

Ciao, Duncan.

Duncan,
I don't believe that this is due to symbols defined in libcloog itself because I see this same
issue if I build gcc-fsf-4.7 using --without-cloog and --without-ppl to disable graphite support
such that cc1 is linked as...

/sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1:
  /sw/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
  /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.1.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

It is interesting though that LLVMPolly.so shows the linkage...

/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

compared to how isl/cloog/gmp is linked in gcc 4.8...

/sw/lib/gcc4.8/libexec/gcc/x86_64-apple-darwin12.2.0/4.8.0/cc1:
  /sw/lib/libintl.8.dylib (compatibility version 10.0.0, current version 10.1.0)
  /sw/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/libmpc.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.1.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.5)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

Shouldn't llvm-3.2/build/tools/polly/CMakeFiles/LLVMPolly.dir/link.txt have...

/sw/lib/libcloog-isl.dylib /sw/lib/libisl.dylib /sw/lib/libgmp.dylib

as the linkage order instead? Unfortunately this change alone doesn't eliminate the
"lazy symbol binding failed: fast lazy bind offset out of range" dyld errors.

However I have managed to shift the error with the following linkage...

LLVMPolly.so:
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmpxx.4.dylib (compatibility version 7.0.0, current version 7.5.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

using "/sw/lib/libcloog-isl.dylib /sw/lib/libisl.dylib /sw/lib/libgmpxx.dylib /sw/lib/libgmp.dylib"
as the final items in link.txt. Note that -pthread is not necessary on darwin as libpthread.dylib
is just a symlink to libSystem.dylib. This linkage shifts the failure to...

% gcc-fsf-4.7 -fplugin=/sw/lib/gcc4.7/lib/dragonegg.so -fplugin-arg-dragonegg-llvm-option=-load:/sw/opt/llvm-3.2/lib/LLVMPolly.so -O3 himenoBMTxpa.c
himenoBMTxpa.c: In function 'main':
himenoBMTxpa.c:79:5: warning: incompatible implicit declaration of built-in function 'strcpy' [enabled by default]
himenoBMTxpa.c: In function 'set_param':
himenoBMTxpa.c:226:5: warning: incompatible implicit declaration of built-in function 'exit' [enabled by default]
himenoBMTxpa.c: In function 'newMat':
himenoBMTxpa.c:239:5: warning: incompatible implicit declaration of built-in function 'malloc' [enabled by default]
himenoBMTxpa.c: In function 'clearMat':
himenoBMTxpa.c:248:5: warning: incompatible implicit declaration of built-in function 'free' [enabled by default]
dyld: lazy symbol binding failed: BIND_OPCODE_SET_SEGMENT_AND_OFFSET_ULEB has segment 5 which is too large (4)

dyld: BIND_OPCODE_SET_SEGMENT_AND_OFFSET_ULEB has segment 5 which is too large (4)

gcc-fsf-4.7: internal compiler error: Trace/BPT trap: 5 (program cc1)
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.

Duncan_Sands · October 23, 2012, 1:58pm

Hi Jack,

It is interesting though that LLVMPolly.so shows the linkage...

/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

...

However I have managed to shift the error with the following linkage...

LLVMPolly.so:
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmpxx.4.dylib (compatibility version 7.0.0, current version 7.5.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

why in this second case do you see libcloog (not present in the first) and two
copies of libgmp?

Ciao, Duncan.

Jack_Howarth1 · October 23, 2012, 2:16pm

Hi Jack,

It is interesting though that LLVMPolly.so shows the linkage...

/sw/opt/llvm-3.2/lib/LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

...

However I have managed to shift the error with the following linkage...

LLVMPolly.so:
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmpxx.4.dylib (compatibility version 7.0.0, current version 7.5.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

why in this second case do you see libcloog (not present in the first) and two
copies of libgmp?

My mistake, I grabbed the wrong lines. The stock build of LLVMPolly shows...

LLVMPolly.so:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)

which is different from what FSF gcc uses. Shouldn't the libraries be ordered so that if library A
is linked against library B that libA should appear before libB? Since on darwin, we have...

/sw/lib/libisl.10.dylib:
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

/sw/lib/libcloog-isl.3.dylib:
  /sw/lib/libcloog-isl.3.dylib (compatibility version 4.0.0, current version 4.0.0)
  /sw/lib/libisl.10.dylib (compatibility version 11.0.0, current version 11.0.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

/sw/lib/gmp5/libgmp.10.dylib:
/sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

I would expect the linkage to be "-lcloog-isl -lisl -lgmp' rather than the current
'-lisl -lcloog-isl -lgmp', no?
As to the linkage of /sw/lib/gmp5/libgmpxx.4.dylib, that was a pure guess. Since we are
using a c++ compiler to compile LLVMPolly, I thought that we might needs some calls from the libgmpxx
library linked it. It is very curious that this linkage shift the error to...

dyld: lazy symbol binding failed: BIND_OPCODE_SET_SEGMENT_AND_OFFSET_ULEB has segment 5 which is too large (4)

dyld: BIND_OPCODE_SET_SEGMENT_AND_OFFSET_ULEB has segment 5 which is too large (4)

gcc-fsf-4.7: internal compiler error: Trace/BPT trap: 5 (progra[prrg4:~] howarth% himenoBMTxpa.c: In function 'main':

I can't find much on that error via google. In any case, if the library contains only unused symbols and
didn't require additional linkages...

/sw/lib/gmp5/libgmpxx.4.dylib:
  /sw/lib/gmp5/libgmpxx.4.dylib (compatibility version 7.0.0, current version 7.5.0)
  /sw/lib/gmp5/libgmp.10.dylib (compatibility version 11.0.0, current version 11.5.0)
  /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 56.0.0)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 169.3.0)

I would have though it should have no effect on the bug. I find this interesting as the addition of
/sw/lib/gmp5/libgmpxx.4.dylib to the linkage was the first to shift the error way from
"lazy symbol binding failed: fast lazy bind offset out of range" to something else.
Note that just correcting the linkage order of libisl/libcloog-isl/libgmp doesn't have
any effect on the "lazy symbol binding failed: fast lazy bind offset out of range" dyld error.
Jack

Jack_Howarth1 · October 23, 2012, 7:50pm

Nick,
Can you clarify what I should be looking for here? I assumed you expected that I should see some symbol
bindings occur immediately after the initializer function in LLVMPolly.so is called but before dyld throws
the "fast lazy bind offset out of range" error message. In the http://llvm.org/bugs/attachment.cgi?id=9397,
the log file for a walk with DYLD_PRINT_INITIALIZERS and DYLD_PRINT_BINDINGS set on darwin12, I only see...

(gdb)
llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x141328518 "/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6d0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
77 void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
(gdb)
Reading symbols for shared libraries ... done
dyld: bind: libisl.10.dylib:0x101292010 = libsystem_c.dylib:__DefaultRuneLocale, *0x101292010 = 0x7FFF7A145890
dyld: bind: libisl.10.dylib:0x101292018 = libsystem_c.dylib:___stack_chk_guard, *0x101292018 = 0x7FFF7A1474B0
....
dyld: weak bind: LLVMPolly.so:0x100FB3128 = libc++.1.dylib:__ZdaPv, *0x100FB3128 = 0x7FFF898BD1C6
dyld: weak bind: LLVMPolly.so:0x100FB3130 = libc++.1.dylib:__ZdlPv, *0x100FB3130 = 0x7FFF898BD1AB
dyld: weak bind: LLVMPolly.so:0x100FB3138 = libc++.1.dylib:__Znam, *0x100FB3138 = 0x7FFF898BD173
dyld: weak bind: LLVMPolly.so:0x100FB3140 = libc++.1.dylib:__Znwm, *0x100FB3140 = 0x7FFF898BD0DF
dyld: calling initializer function 0x100ebb5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00007fff5fc0109d in __dyld_dyld_fatal_error ()

where all of the bindings from LLVMPolly.so occur before the initializer function is called. Was this behavior
expected?
Also, as I noted on http://llvm.org/bugs/show_bug.cgi?id=14140#c13, a static build of LLVMPolly.so on 10.6 exhibits the
same dyld error. Am I correct to assume that test proves that the offending symbol can't be from libcloog-isl, libisl,
libgmp or libstdc++ because, when statically linked, those symbols can't be obtained from cc1? Is it vaild to assume that the
offending symbol has to be both in the binding list seen when LLVMPolly.so loads and the symbols (defined or undefined)
in cc1? Any suggestions on how to proceed in debugging this would be appreciated.
Jack

Nick_Kledzik · October 23, 2012, 8:05pm

As I said before, run once to get the address of the initializer being run inside the dlopen call (0x100ebb5b0 in the case above). Set a break point there. Run again. When it stops at the break point (inside the initializer), start stepping until it crashes. You may need to step at the instruction level.

The point of the DYLD_PRINT_BINDINGS was so you can see what pointers dyld was setting. At some point in your trace you may be jumping through one of those pointers. You can check that the value makes sense and was not stomped by something else.

-Nick

Jack_Howarth1 · October 23, 2012, 8:57pm

>>
>>
>>> Nick,
>>> I have uploaded the full walk with 'set env DYLD_PRINT_INITIALIZERS'. It didn't seem very informative
>>> as the dyld error occurs right after...
>>>
>>> (gdb)
>>> llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x142903da8 "/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6e0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
>>> 77 void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
>>> (gdb)
>>> Reading symbols for shared libraries ... done
>>> dyld: calling initializer function 0x100eea5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
>>
>> You need to set a break point at 0x100eea5b0 and run again until that break is hit, then start stepping.
>>
>>> Are there any tricks for identifying the order that the symbols are being resolved?
>> (gdb) set env DYLD_PRINT_BINDINGS 1
>> will show each use of a symbol as it is bound. This can be *a lot* of information.
>>
>> -Nick
>
> Nick,
> Can you clarify what I should be looking for here? I assumed you expected that I should see some symbol
> bindings occur immediately after the initializer function in LLVMPolly.so is called but before dyld throws
> the "fast lazy bind offset out of range" error message. In the http://llvm.org/bugs/attachment.cgi?id=9397,
> the log file for a walk with DYLD_PRINT_INITIALIZERS and DYLD_PRINT_BINDINGS set on darwin12, I only see...
>
> (gdb)
> llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x141328518 "/sw/opt/llvm-3.2/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6d0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
> 77 void *handle = dlopen(filename, RTLD_LAZY|RTLD_GLOBAL);
> (gdb)
> Reading symbols for shared libraries ... done
> dyld: bind: libisl.10.dylib:0x101292010 = libsystem_c.dylib:__DefaultRuneLocale, *0x101292010 = 0x7FFF7A145890
> dyld: bind: libisl.10.dylib:0x101292018 = libsystem_c.dylib:___stack_chk_guard, *0x101292018 = 0x7FFF7A1474B0
> ....
> dyld: weak bind: LLVMPolly.so:0x100FB3128 = libc++.1.dylib:__ZdaPv, *0x100FB3128 = 0x7FFF898BD1C6
> dyld: weak bind: LLVMPolly.so:0x100FB3130 = libc++.1.dylib:__ZdlPv, *0x100FB3130 = 0x7FFF898BD1AB
> dyld: weak bind: LLVMPolly.so:0x100FB3138 = libc++.1.dylib:__Znam, *0x100FB3138 = 0x7FFF898BD173
> dyld: weak bind: LLVMPolly.so:0x100FB3140 = libc++.1.dylib:__Znwm, *0x100FB3140 = 0x7FFF898BD0DF
> dyld: calling initializer function 0x100ebb5b0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
> dyld: lazy symbol binding failed: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
> dyld: fast lazy bind offset out of range (53437, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
>
> Program received signal SIGTRAP, Trace/breakpoint trap.
> 0x00007fff5fc0109d in __dyld_dyld_fatal_error ()
>
> where all of the bindings from LLVMPolly.so occur before the initializer function is called. Was this behavior
> expected?
> Also, as I noted on http://llvm.org/bugs/show_bug.cgi?id=14140#c13, a static build of LLVMPolly.so on 10.6 exhibits the
> same dyld error. Am I correct to assume that test proves that the offending symbol can't be from libcloog-isl, libisl,
> libgmp or libstdc++ because, when statically linked, those symbols can't be obtained from cc1? Is it vaild to assume that the
> offending symbol has to be both in the binding list seen when LLVMPolly.so loads and the symbols (defined or undefined)
> in cc1? Any suggestions on how to proceed in debugging this would be appreciated.

As I said before, run once to get the address of the initializer being run inside the dlopen call (0x100ebb5b0 in the case above). Set a break point there. Run again. When it stops at the break point (inside the initializer), start stepping until it crashes. You may need to step at the instruction level.

The point of the DYLD_PRINT_BINDINGS was so you can see what pointers dyld was setting. At some point in your trace you may be jumping through one of those pointers. You can check that the value makes sense and was not stomped by something else.

-Nick

Nick,
Can I do this without access to a debug version of dyld? Using the copy of LLVMPolly.so with isl/cloog-isl/gmp statically linked,
I find that if I set the breakpoint to the address of the initializer...

dyld: calling initializer function 0x100ebb3a0 in /sw/opt/llvm-3.2/lib/LLVMPolly.so
dyld: lazy symbol binding failed: fast lazy bind offset out of range (114808, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1
dyld: fast lazy bind offset out of range (114808, max=2928) in image /sw/lib/gcc4.7/libexec/gcc/x86_64-apple-darwin12.2.0/4.7.2/cc1

with...

(gdb) break *0x100ebb3a0
Breakpoint 2 at 0x100ebb3a0

this lands me at...

dyld: weak bind: LLVMPolly.so:0x1010F4BD0 = libc++.1.dylib:__Znwm, *0x1010F4BD0 = 0x7FFF898BD0DF

Breakpoint 2, 0x0000000100ebb3a0 in pch_address_space ()
(gdb) si
0x0000000100ebb3a1 in pch_address_space ()
(gdb) si
0x0000000100ebb3a4 in pch_address_space ()
(gdb) si
0x0000000100ebb380 in pch_address_space ()
(gdb) si
0x0000000100ebb381 in pch_address_space ()
...
and on in various dyld calls. Will I really be able to get anything useful from this without a debug build of the system
dyld installed?
Jack

Nick_Kledzik · October 23, 2012, 9:03pm

You just need a build of LLVMPolly.so that does not have symbols stripped. Or get the .dSYM file for your build of LLVMPolly.so and load that in gdb.

-Nick

Jack_Howarth1 · October 23, 2012, 11:46pm

Nick,
I have uploaded a bzip2 compressed log of the 'si' walk from the failing initializer in LLVMPolly.so
to http://llvm.org/bugs/attachment.cgi?id=9408 in http://llvm.org/bugs/show_bug.cgi?id=14140. The backtrace
from the final crash point shows...

(gdb) bt
#0 0x00007fff5fc0123f in __dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
#1 0x00007fff5fc02138 in __dyld__ZN4dyld4haltEPKc ()
#2 0x00007fff5fc04048 in __dyld__ZN4dyld18fastBindLazySymbolEPP11ImageLoaderm ()
#3 0x00007fff8bd808ee in dyld_stub_binder_ ()
#4 0x0000000100faf3e0 in Json::Value::maxUInt ()
#5 0x0000000100ebad65 in pch_address_space ()
#6 0x0000000100ebb5a0 in pch_address_space ()
#7 0x0000000100ebb5b9 in pch_address_space ()
#8 0x00007fff5fc13378 in __dyld__ZN16ImageLoaderMachO18doModInitFunctionsERKN11ImageLoader11LinkContextE ()
#9 0x00007fff5fc13762 in __dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkContextE ()
#10 0x00007fff5fc1006e in __dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEjRNS_21InitializerTimingListE ()
#11 0x00007fff5fc0feba in __dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextERNS_21InitializerTimingListE ()
#12 0x00007fff5fc04e38 in __dyld__ZN4dyld15runInitializersEP11ImageLoader ()
#13 0x00007fff5fc0a87c in __dyld_dlopen ()
#14 0x00007fff8bd81dd8 in dlopen ()
#15 0x0000000142f9820f in llvm::sys::DynamicLibrary::getPermanentLibrary (filename=0x141328d38 "/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", errMsg=0x7fff5fbfe6a0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/DynamicLibrary.cpp:77
#16 0x0000000142f79ced in llvm::sys::DynamicLibrary::LoadLibraryPermanently (Filename=0x141328d38 "/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", ErrMsg=0x7fff5fbfe6a0) at DynamicLibrary.h:77
#17 0x0000000142f79ab9 in llvm::PluginLoader::operator= (this=0x143497418, Filename=@0x7fff5fbfe780) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/PluginLoader.cpp:29
#18 0x00000001422137de in llvm::opt_storage<llvm::PluginLoader, false, true>::setValue<std::string> (this=0x143497418, V=@0x7fff5fbfe780, initial=false) at CommandLine.h:1072
#19 0x0000000142213271 in llvm::opt<llvm::PluginLoader, false, llvm::parser<std::string> >::handleOccurrence (this=0x1434973e0, pos=2, ArgName={Data = 0x1413259f1 "load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 4, static npos = 18446744073709551615}, Arg={Data = 0x1413259f6 "/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 63, static npos = 18446744073709551615}) at CommandLine.h:1131
#20 0x0000000142f5e731 in llvm::Option::addOccurrence (this=0x1434973e0, pos=2, ArgName={Data = 0x1413259f1 "load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 4, static npos = 18446744073709551615}, Value={Data = 0x1413259f6 "/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 63, static npos = 18446744073709551615}, MultiArg=false) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:883
#21 0x0000000142f635ab in CommaSeparateAndAddOccurence (Handler=0x1434973e0, pos=2, ArgName={Data = 0x1413259f1 "load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 4, static npos = 18446744073709551615}, Value={Data = 0x1413259f6 "/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 63, static npos = 18446744073709551615}, MultiArg=false) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:259
#22 0x0000000142f5ea31 in ProvideOption (Handler=0x1434973e0, ArgName={Data = 0x1413259f1 "load=/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 4, static npos = 18446744073709551615}, Value={Data = 0x1413259f6 "/sw/src/fink.build/llvm32-3.2-0/llvm-3.2/build/lib/LLVMPolly.so", Length = 63, static npos = 18446744073709551615}, argc=4, argv=0x141325a40, i=@0x7fff5fbfefb0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:299
#23 0x0000000142f5c7d4 in llvm::ParseCommandLineOptions (argc=4, argv=0x141325a40, Overview=0x0) at /sw/src/fink.build/llvm32-3.2-0/llvm-3.2/lib/Support/CommandLine.cpp:724
#24 0x0000000142209df1 in ConfigureLLVM () at /sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/src/Backend.cpp:372
#25 0x0000000142208762 in InitializeBackend () at /sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/src/Backend.cpp:583
#26 0x00000001422079a8 in llvm_emit_globals () at /sw/src/fink.build/dragonegg-gcc47-3.2-0/dragonegg-3.2/src/Backend.cpp:1741
(gdb)

The crash shows up as in the walk as...

(gdb)
0x00007fff5fc24182 in __dyld_mach_init ()
(gdb)
0x00007fff5fc01239 in __dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
(gdb)
0x00007fff5fc0123c in __dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
(gdb)
0x00007fff5fc0123f in __dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()
(gdb)

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000008
0x00007fff5fc0123f in __dyld__ZN13dyldbootstrap5startEPK12macho_headeriPPKclS2_Pm ()

The llvm build is built as a debug build as is the dragonegg build so both have been
built with -g and are unstripped. I am still very unclear on how I am supposed to
extract the offending symbol from this log. Since dyld seems to be from reading pch,
I assume this is a symbol from libstdc++. This would seem odd since cc1 isn't built
with c++ on gcc 4.7.2 and isn't linked against libstdc++. The dragonegg plugin that
dlopens the LLVMPolly plugin is built with c+ and linked against libstdc++ but I
verified that building it with the c compiler to avoid the libstdc++ linkage doesn't
solve this bug. Thanks in advance for any analysis of the log.
Jack