lli: problem finding external symbol

Hi, I have a large-ish application I'm trying to build with LLVM, so I
replaced g++ in the Makefiles with llvm-g++. This appears to work fine
when I pass -Wl,-native to generate an ELF executable, but I want to
control the inlining threshold it uses, so I am now passing
-Wl,-disable-inlining and -Wa,-disable-inlining, and having it
generate LLVM bytecode, then running opt -inlining to generate
versions with different thresholds. However, from there I can't run
the bytecode with lli - I get the following error:

$ lli -load=/home/mmccrack/linux/lib/libTau.sh leo-inline-300-tau.bc
ERROR: Program used external function
'__llvm_cxxeh_allocate_exception' which could not be resolved!
lli((anonymous namespace)::PrintStackTrace()+0x18)[0x8629a74]
lli((anonymous namespace)::SignalHandler(int)+0xfe)[0x8629cfc]
/lib/tls/libc.so.6[0x4d9eb8]
/lib/tls/libc.so.6(abort+0x1d5)[0x4db4e5]
lli(llvm::JIT::getPointerToNamedFunction(std::basic_string<char,
std::char_traits<char>, std::allocator<char> >
const&)+0xc2)[0x8342394]
...skip rest of bt...

I found that symbol in the bytecode version of libstdc++, which the
disassembled llvm code does show as a dependent library:

; ModuleID = 'leo-inline-300-tau.bc'
target endian = little
target pointersize = 32
target triple = "i686-pc-linux-gnu"
deplibs = [ "stdc++", "c", "crtend" ]

Furthermore, the path to that library is in my LLVM_LIB_SEARCH_PATHS
variable, even though the docs say that's not strictly necessary.

An important point here is that I'm using a source-to-source
instrumentation toolkit (TAU) to insert instrumentation before I
compile with llvm-g++. When I don't use it, llvm-g++ compiles a
working bytecode file. However, the only thing I can see that's
different is that the code has added some references to an
instrumentation library, which I can load just fine, as above.

Note: in more recent tests, it fails to find the symbol
_ZN9__gnu_cxx12__pool_allocILb1ELi0EE8allocateEj instead, so it's not
specific to the above symbol.

Thanks for any help you can give,
-mike

Appendix: Here are sample compilation commands for building, first
just using LLVM and second using the source-to-source instrumentation
tool.

/home/mmccrack/lens/cfrontend/x86/llvm-gcc/bin/llvm-g++ -g
-I/usr/X11R6/include -I. -I/home/mmccrack/include -Icore -Igeometry
-Iimage -Iimage/sky -Iimage/tm -Inet -Isampling -Ishaders
-Ithread -Itools -Iparse -Iparse -m32 -fpermissive -fno-rtti
-pthread -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_LARGEFILE_SOURCE
-D_GNU_SOURCE -O3 -march=pentium3 -U__USE_ASM__ -U__USE_SSE_
-U__USE_SSE2__ -MD -c -o .o/core/leoLight.o core/leoLight.cpp

And now the monstrosity:

/home/mmccrack/linux/bin/tau_compiler.sh
-optPdtDir="/home/mmccrack//linux"
-optPdtCOpts="-I/home/mmccrack/include -DPROFILING_ON
-DTAU_STDCXXLIB -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS -fPIC
    -DPTHREADS
                                                -DTAU_LARGEFILE
-D_LARGEFILE64_SOURCE "
-optPdtCxxOpts="-I/home/mmccrack/include -DPROFILING_ON
             -DTAU_STDCXXLIB -DTAU_GNU
-DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS
                              -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE "
-optTauInstr="/home/mmccrack/linux/bin/tau_instrumentor" -optNoMpi
-optOpariDir="" -optOpariTool="" -optTauCC="gcc "
-optCompile="-I/home/mmccrack/include -DPROFILING_ON
                 -DTAU_STDCXXLIB -DTAU_GNU
-DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS
-DTAU_LARGEFILE -D_LARGEFILE64_SOURCE " -optLinking="
-L/home/mmccrack/linux/lib -ltau-pthread-pdt
            -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ -lstdc++ -lgcc_s
" -optKeepFiles -optTauSelectFile=tau_select_file
/home/mmccrack/lens/cfrontend/x86/llvm-gcc/bin/llvm-g++ -g
-I/home/mmccrack/include -DPROFILING_ON
-DTAU_STDCXXLIB -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS -fPIC
   -DPTHREADS -DTAU_LARGEFILE
-D_LARGEFILE64_SOURCE -I/usr/X11R6/include -I.
-I/home/mmccrack/include -Icore -Igeometry -Iimage -Iimage/sky
-Iimage/tm -Inet -Isampling -Ishaders -Ithread -Itools -Iparse
-Iparse -m32 -fpermissive -fno-rtti -pthread -D_FILE_OFFSET_BITS=64
-D_REENTRANT -D_LARGEFILE_SOURCE -D_GNU_SOURCE -O3 -march=pentium3
-L/home/mmccrack/linux/lib -ltau-pthread-pdt
-U__USE_ASM__ -U__USE_SSE_ -U__USE_SSE2__ -MD -c -o
.o/thread/threadbase.o thread/threadbase.cpp

Note that many things are repeated in quotes, and TAU_STDCXXLIB just
tells its headers to use <vector> instead of <vector.h>...

the bytecode with lli - I get the following error:

$ lli -load=/home/mmccrack/linux/lib/libTau.sh leo-inline-300-tau.bc
ERROR: Program used external function
'__llvm_cxxeh_allocate_exception' which could not be resolved!

This is most likely because you are trying to link with llvmgcc, not with llvm-g++. Make sure that the problem is linked with llvm-g++ and this should be fixed.

-Chris

lli((anonymous namespace)::PrintStackTrace()+0x18)[0x8629a74]
lli((anonymous namespace)::SignalHandler(int)+0xfe)[0x8629cfc]
/lib/tls/libc.so.6[0x4d9eb8]
/lib/tls/libc.so.6(abort+0x1d5)[0x4db4e5]
lli(llvm::JIT::getPointerToNamedFunction(std::basic_string<char,
std::char_traits<char>, std::allocator<char> >
const&)+0xc2)[0x8342394]
...skip rest of bt...

I found that symbol in the bytecode version of libstdc++, which the
disassembled llvm code does show as a dependent library:

; ModuleID = 'leo-inline-300-tau.bc'
target endian = little
target pointersize = 32
target triple = "i686-pc-linux-gnu"
deplibs = [ "stdc++", "c", "crtend" ]

Furthermore, the path to that library is in my LLVM_LIB_SEARCH_PATHS
variable, even though the docs say that's not strictly necessary.

An important point here is that I'm using a source-to-source
instrumentation toolkit (TAU) to insert instrumentation before I
compile with llvm-g++. When I don't use it, llvm-g++ compiles a
working bytecode file. However, the only thing I can see that's
different is that the code has added some references to an
instrumentation library, which I can load just fine, as above.

Note: in more recent tests, it fails to find the symbol
_ZN9__gnu_cxx12__pool_allocILb1ELi0EE8allocateEj instead, so it's not
specific to the above symbol.

Thanks for any help you can give,
-mike

Appendix: Here are sample compilation commands for building, first
just using LLVM and second using the source-to-source instrumentation
tool.

/home/mmccrack/lens/cfrontend/x86/llvm-gcc/bin/llvm-g++ -g
-I/usr/X11R6/include -I. -I/home/mmccrack/include -Icore -Igeometry
-Iimage -Iimage/sky -Iimage/tm -Inet -Isampling -Ishaders
-Ithread -Itools -Iparse -Iparse -m32 -fpermissive -fno-rtti
-pthread -D_FILE_OFFSET_BITS=64 -D_REENTRANT -D_LARGEFILE_SOURCE
-D_GNU_SOURCE -O3 -march=pentium3 -U__USE_ASM__ -U__USE_SSE_
-U__USE_SSE2__ -MD -c -o .o/core/leoLight.o core/leoLight.cpp

And now the monstrosity:

/home/mmccrack/linux/bin/tau_compiler.sh
-optPdtDir="/home/mmccrack//linux"
-optPdtCOpts="-I/home/mmccrack/include -DPROFILING_ON
-DTAU_STDCXXLIB -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS -fPIC
   -DPTHREADS
                                               -DTAU_LARGEFILE
-D_LARGEFILE64_SOURCE "
-optPdtCxxOpts="-I/home/mmccrack/include -DPROFILING_ON
            -DTAU_STDCXXLIB -DTAU_GNU
-DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS
                             -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE "
-optTauInstr="/home/mmccrack/linux/bin/tau_instrumentor" -optNoMpi
-optOpariDir="" -optOpariTool="" -optTauCC="gcc "
-optCompile="-I/home/mmccrack/include -DPROFILING_ON
                -DTAU_STDCXXLIB -DTAU_GNU
-DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS
-DTAU_LARGEFILE -D_LARGEFILE64_SOURCE " -optLinking="
-L/home/mmccrack/linux/lib -ltau-pthread-pdt
           -L/usr/lib/gcc-lib/i386-pc-linux/3.2.3/ -lstdc++ -lgcc_s
" -optKeepFiles -optTauSelectFile=tau_select_file
/home/mmccrack/lens/cfrontend/x86/llvm-gcc/bin/llvm-g++ -g
-I/home/mmccrack/include -DPROFILING_ON
-DTAU_STDCXXLIB -DTAU_GNU -DTAU_DOT_H_LESS_HEADERS -fPIC
  -DPTHREADS -DTAU_LARGEFILE
-D_LARGEFILE64_SOURCE -I/usr/X11R6/include -I.
-I/home/mmccrack/include -Icore -Igeometry -Iimage -Iimage/sky
-Iimage/tm -Inet -Isampling -Ishaders -Ithread -Itools -Iparse
-Iparse -m32 -fpermissive -fno-rtti -pthread -D_FILE_OFFSET_BITS=64
-D_REENTRANT -D_LARGEFILE_SOURCE -D_GNU_SOURCE -O3 -march=pentium3
-L/home/mmccrack/linux/lib -ltau-pthread-pdt
-U__USE_ASM__ -U__USE_SSE_ -U__USE_SSE2__ -MD -c -o
.o/thread/threadbase.o thread/threadbase.cpp

Note that many things are repeated in quotes, and TAU_STDCXXLIB just
tells its headers to use <vector> instead of <vector.h>...

--
Michael McCracken
UCSD CSE PhD Candidate
research: http://www.cse.ucsd.edu/~mmccrack/
misc: http://michael-mccracken.net/wp/

_______________________________________________
LLVM Developers mailing list
LLVMdev@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-Chris

No, it is definitely using llvm-g++ - the link command isn't different
between the instrumented version and the uninstrumented version, which
works. And stdc++ is listed as one of the dependent libraries in the
generated bytecode.

-mike