issues with split llvm libraries and llvmpipe and failing to load library

Hey,

So in Fedora rawhide we are now building llvm 3.7.1 into the lots of
little shared libraries format.

However I'm running into a major problem with the fact that sometimes
dlclose isn't dropping all the LLVM libraries from the address space
of the process.

We have a sequence like this:

a) X server asks mesa gbm library to init, it loads the
kms_swrast_dri.so with dlopen(LAZY|GLOBAL). kms_swrast_dri.so is
linked against a large bunch of LLVM libraries (see below).

b) gbm discovers it can't do what it wants and dlcloses the library.
At this point a bunch of the LLVM libraries drop out of the map,
pretty much everything down to LLVMTarget. Everything from
LLVMTarget onwards remains loaded and I've no idea how to discover why.

c) later X tries to load kms_swrast_dri.so for GLX usage, and it
brings back in all the LLVM libraries that got dropped out, however as
LLVMObject has never been cleaned up, it has all the command line
options in it, so we get
: CommandLine Error: Option 'x86-asm-syntax' registered more than once!
LLVM ERROR: inconsistency in registered CommandLine options
and things crash.

So anyone any ideas why we the linker isn't dropping all the other
LLVM libraries, or any workaround for this, apart from just going back
to the single libLLVM?

Dave.

[airlied@f21vm ~]$ ldd /usr/lib64/dri/kms_swrast_dri.so
    linux-vdso.so.1 (0x00007ffec17fe000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f8a68ac5000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a688a3000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a68685000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a68481000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f8a68255000)
    libdrm_intel.so.1 => /lib64/libdrm_intel.so.1 (0x00007f8a68033000)
    libdrm_nouveau.so.2 => /lib64/libdrm_nouveau.so.2 (0x00007f8a67e2b000)
    libdrm_radeon.so.1 => /lib64/libdrm_radeon.so.1 (0x00007f8a67c1f000)
    libdrm_amdgpu.so.1 => /lib64/libdrm_amdgpu.so.1 (0x00007f8a67a16000)
    libdrm.so.2 => /lib64/libdrm.so.2 (0x00007f8a67807000)
    libelf.so.1 => /lib64/libelf.so.1 (0x00007f8a675ef000)
    libLLVMAMDGPUCodeGen.so.3.7 => /lib64/libLLVMAMDGPUCodeGen.so.3.7
(0x00007f8a672e2000)
    libLLVMAMDGPUAsmParser.so.3.7 =>
/lib64/libLLVMAMDGPUAsmParser.so.3.7 (0x00007f8a670a1000)
    libLLVMAMDGPUUtils.so.3.7 => /lib64/libLLVMAMDGPUUtils.so.3.7
(0x00007f8a66e9f000)
    libLLVMAMDGPUDesc.so.3.7 => /lib64/libLLVMAMDGPUDesc.so.3.7
(0x00007f8a66b9b000)
    libLLVMAMDGPUInfo.so.3.7 => /lib64/libLLVMAMDGPUInfo.so.3.7
(0x00007f8a66999000)
    libLLVMAMDGPUAsmPrinter.so.3.7 =>
/lib64/libLLVMAMDGPUAsmPrinter.so.3.7 (0x00007f8a66774000)
    libLLVMObjCARCOpts.so.3.7 => /lib64/libLLVMObjCARCOpts.so.3.7
(0x00007f8a6654e000)
    libLLVMOption.so.3.7 => /lib64/libLLVMOption.so.3.7 (0x00007f8a66340000)
    libLLVMIRReader.so.3.7 => /lib64/libLLVMIRReader.so.3.7 (0x00007f8a6613b000)
    libLLVMAsmParser.so.3.7 => /lib64/libLLVMAsmParser.so.3.7
(0x00007f8a65ef2000)
    libLLVMLinker.so.3.7 => /lib64/libLLVMLinker.so.3.7 (0x00007f8a65cdc000)
    libLLVMipo.so.3.7 => /lib64/libLLVMipo.so.3.7 (0x00007f8a65a4a000)
    libLLVMVectorize.so.3.7 => /lib64/libLLVMVectorize.so.3.7
(0x00007f8a657ef000)
    libLLVMAArch64Disassembler.so.3.7 =>
/lib64/libLLVMAArch64Disassembler.so.3.7 (0x00007f8a655d0000)
    libLLVMAArch64CodeGen.so.3.7 =>
/lib64/libLLVMAArch64CodeGen.so.3.7 (0x00007f8a652b0000)
    libLLVMAArch64AsmParser.so.3.7 =>
/lib64/libLLVMAArch64AsmParser.so.3.7 (0x00007f8a65058000)
    libLLVMAArch64Desc.so.3.7 => /lib64/libLLVMAArch64Desc.so.3.7
(0x00007f8a64dde000)
    libLLVMAArch64Info.so.3.7 => /lib64/libLLVMAArch64Info.so.3.7
(0x00007f8a64bdc000)
    libLLVMAArch64AsmPrinter.so.3.7 =>
/lib64/libLLVMAArch64AsmPrinter.so.3.7 (0x00007f8a64979000)
    libLLVMAArch64Utils.so.3.7 => /lib64/libLLVMAArch64Utils.so.3.7
(0x00007f8a64768000)
    libLLVMBitWriter.so.3.7 => /lib64/libLLVMBitWriter.so.3.7
(0x00007f8a64541000)
    libLLVMX86Disassembler.so.3.7 =>
/lib64/libLLVMX86Disassembler.so.3.7 (0x00007f8a641df000)
    libLLVMX86AsmParser.so.3.7 => /lib64/libLLVMX86AsmParser.so.3.7
(0x00007f8a63f4b000)
    libLLVMX86CodeGen.so.3.7 => /lib64/libLLVMX86CodeGen.so.3.7
(0x00007f8a63b61000)
    libLLVMSelectionDAG.so.3.7 => /lib64/libLLVMSelectionDAG.so.3.7
(0x00007f8a63761000)
    libLLVMAsmPrinter.so.3.7 => /lib64/libLLVMAsmPrinter.so.3.7
(0x00007f8a634e6000)
    libLLVMCodeGen.so.3.7 => /lib64/libLLVMCodeGen.so.3.7 (0x00007f8a63014000)
    libLLVMScalarOpts.so.3.7 => /lib64/libLLVMScalarOpts.so.3.7
(0x00007f8a62c92000)
    libLLVMProfileData.so.3.7 => /lib64/libLLVMProfileData.so.3.7
(0x00007f8a62a67000)
    libLLVMInstCombine.so.3.7 => /lib64/libLLVMInstCombine.so.3.7
(0x00007f8a627aa000)
    libLLVMInstrumentation.so.3.7 =>
/lib64/libLLVMInstrumentation.so.3.7 (0x00007f8a6253f000)
    libLLVMTransformUtils.so.3.7 =>
/lib64/libLLVMTransformUtils.so.3.7 (0x00007f8a6226b000)
    libLLVMipa.so.3.7 => /lib64/libLLVMipa.so.3.7 (0x00007f8a62043000)
    libLLVMX86Desc.so.3.7 => /lib64/libLLVMX86Desc.so.3.7 (0x00007f8a61c95000)
    libLLVMMCDisassembler.so.3.7 =>
/lib64/libLLVMMCDisassembler.so.3.7 (0x00007f8a61a8f000)
    libLLVMX86Info.so.3.7 => /lib64/libLLVMX86Info.so.3.7 (0x00007f8a6188d000)
    libLLVMX86AsmPrinter.so.3.7 => /lib64/libLLVMX86AsmPrinter.so.3.7
(0x00007f8a61647000)
    libLLVMX86Utils.so.3.7 => /lib64/libLLVMX86Utils.so.3.7 (0x00007f8a6143e000)
    libLLVMMCJIT.so.3.7 => /lib64/libLLVMMCJIT.so.3.7 (0x00007f8a61233000)
    libLLVMExecutionEngine.so.3.7 =>
/lib64/libLLVMExecutionEngine.so.3.7 (0x00007f8a61011000)
    libLLVMTarget.so.3.7 => /lib64/libLLVMTarget.so.3.7 (0x00007f8a60e02000)
    libLLVMAnalysis.so.3.7 => /lib64/libLLVMAnalysis.so.3.7 (0x00007f8a60a33000)
    libLLVMRuntimeDyld.so.3.7 => /lib64/libLLVMRuntimeDyld.so.3.7
(0x00007f8a607e4000)
    libLLVMObject.so.3.7 => /lib64/libLLVMObject.so.3.7 (0x00007f8a60574000)
    libLLVMMCParser.so.3.7 => /lib64/libLLVMMCParser.so.3.7 (0x00007f8a60347000)
    libLLVMBitReader.so.3.7 => /lib64/libLLVMBitReader.so.3.7
(0x00007f8a60112000)
    libLLVMMC.so.3.7 => /lib64/libLLVMMC.so.3.7 (0x00007f8a5fe91000)
    libLLVMCore.so.3.7 => /lib64/libLLVMCore.so.3.7 (0x00007f8a5fa56000)
    libLLVMSupport.so.3.7 => /lib64/libLLVMSupport.so.3.7 (0x00007f8a5f66a000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f8a5f360000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8a5f149000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f8a5ed72000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f8a5eb5c000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a5e8e7000)
    /lib64/ld-linux-x86-64.so.2 (0x0000565263dae000)
    libpciaccess.so.0 => /lib64/libpciaccess.so.0 (0x00007f8a5e6dc000)
    librt.so.1 => /lib64/../lib64/librt.so.1 (0x00007f8a5e4d2000)

Hey,

So in Fedora rawhide we are now building llvm 3.7.1 into the lots of
little shared libraries format.

Barring all the other problems, are you sure this is a good idea? I remember somebody mentioning that at least LLVM's own test suite runs much slower in this build configuration. I don't know whether this is "only" due to a slower startup time in the dynamic linker or also due to slower compile times at runtime, but it's probably worth measuring.

Another thing to keep in mind is that of the many buildbots LLVM has, apparently _none of them_ run in the lots of little shared libraries format (as I discovered a few months ago thanks to a regression there). Unless you can convince the LLVM folks to change that, you (Fedora) will likely end up in the fun situation of becoming the de facto maintainer of that format.

Sorry I can't be of more help with the actual issues you're having.

Nicolai

Hey,

So in Fedora rawhide we are now building llvm 3.7.1 into the lots of
little shared libraries format.

This configuration is only recommended for developers.

See the documentation for BUILD_SHARED_LIBS:BOOL here: http://llvm.org/docs/CMake.html

-Tom

Did you call dlsym at some point? I think this can also increase the refcount, but my memory is fuzzy about the subtleties of dlopen/dlclose.

Interesting I'm fairly sure this wasn't always this way, and at one
point the CMake guys
were saying they should remove the super library.

Ah well live and learn, back to super library it is.

Dave.

So in Fedora rawhide we are now building llvm 3.7.1 into the lots of
little shared libraries format.

Barring all the other problems, are you sure this is a good idea? I
remember somebody mentioning that at least LLVM's own test suite runs
much slower in this build configuration.

I didn't measure the LLVM testsuite wrt this, but I did measure that the
split shared libraries increase the runtime of a single piglit test by
about 0.02s, which for a full piglit run adds up to about 400 CPU core
seconds, i.e. about 6.667 CPU core minutes.

I don't know whether this is "only" due to a slower startup time in
the dynamic linker or also due to slower compile times at runtime, but
it's probably worth measuring.

I didn't investigate it in detail, but I assume it's the former. I don't
know of any other significant difference between the two cases I measured.