Mlir-opt size and link times - optimize?

MLIR
#1

On a typical modern workstation, I notice that mlir-opt takes about 2.3s to link (release assert build), and 8.7s for a debug build. The binary size is 35 MB in the former case and 221 MB in the latter. Here are the top 30 things being linked by size (size in KB) - this is pretty much everything that is over 1 MB in size:

19164 lib/libLLVMCodeGen.a
13404 lib/libLLVMAnalysis.a
12268 lib/libLLVMScalarOpts.a
9608 lib/libLLVMX86CodeGen.a
9136 lib/libLLVMCore.a
8444 lib/libLLVMSelectionDAG.a
8260 lib/libMLIRSPIRV.a
6804 lib/libLLVMipo.a
6552 lib/libLLVMTransformUtils.a
4588 lib/libLLVMSupport.a
4368 lib/libLLVMInstCombine.a
3920 lib/libLLVMX86Desc.a
3748 lib/libMLIRTestDialect.a
3432 lib/libLLVMInstrumentation.a
3060 lib/libLLVMVectorize.a
2880 lib/libLLVMObject.a
2868 lib/libMLIRLLVMIR.a
2604 lib/libLLVMAsmPrinter.a
2512 lib/libMLIRStandardOps.a
2304 lib/libLLVMGlobalISel.a
2292 lib/libLLVMMC.a
2096 lib/libLLVMNVPTXCodeGen.a
2092 lib/libMLIRIR.a
2024 lib/libMLIRSPIRVSerialization.a
2004 lib/libLLVMDebugInfoCodeView.a
1576 lib/libLLVMDebugInfoDWARF.a
1500 lib/libMLIRVector.a
1312 lib/libMLIRTransforms.a
1228 lib/libMLIRLinalgTransforms.a
1188 lib/libMLIRLinalgOps.a

Since mlir-opt itself doesn’t generate LLVM IR nor analyzes/transforms LLVM IR, is there a way to untangle it from many of these? I can’t immediately tell why some of these are even being linked in! Here’s the entire link line for reference:

/usr/lib64/ccache/clang++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -O2 -Wl,-allow-shlib-undefined -Wl,-rpath-link,/home/uday/llvm-project-github/build.release/./lib -Wl,-O3 -Wl,--gc-sections tools/mlir/tools/mlir-opt/CMakeFiles/mlir-opt.dir/mlir-opt.cpp.o -o bin/mlir-opt -Wl,-rpath,"\$ORIGIN/../lib" -lpthread lib/libMLIRAffine.a lib/libMLIRAffineTransforms.a lib/libMLIRAVX512.a lib/libMLIRFxpMathOps.a lib/libMLIRGPU.a lib/libMLIRLinalgAnalysis.a lib/libMLIRLinalgEDSC.a lib/libMLIRLinalgOps.a lib/libMLIRLinalgTransforms.a lib/libMLIRLinalgUtils.a lib/libMLIRLLVMIRTransforms.a lib/libMLIRLLVMIR.a lib/libMLIRLLVMAVX512.a lib/libMLIRNVVMIR.a lib/libMLIRROCDLIR.a lib/libMLIRLoopOps.a lib/libMLIRLoopOpsTransforms.a lib/libMLIROpenMP.a lib/libMLIRQuant.a lib/libMLIRSDBM.a lib/libMLIRShape.a lib/libMLIRSPIRV.a lib/libMLIRSPIRVSerialization.a lib/libMLIRSPIRVTransforms.a lib/libMLIRStandardOps.a lib/libMLIRVector.a lib/libMLIRAffineToStandard.a lib/libMLIRAVX512ToLLVM.a lib/libMLIRGPUtoCUDATransforms.a lib/libMLIRGPUtoNVVMTransforms.a lib/libMLIRGPUtoROCDLTransforms.a lib/libMLIRGPUtoSPIRVTransforms.a lib/libMLIRGPUtoVulkanTransforms.a lib/libMLIRLinalgToLLVM.a lib/libMLIRLinalgToSPIRVTransforms.a lib/libMLIRLoopsToGPU.a lib/libMLIRLoopToStandard.a lib/libMLIRStandardToLLVM.a lib/libMLIRStandardToSPIRVTransforms.a lib/libMLIRStandardToStandard.a lib/libMLIRVectorToLLVM.a lib/libMLIRVectorToLoops.a lib/libMLIRLoopOpsTransforms.a lib/libMLIRLoopAnalysis.a lib/libMLIRAffineTransformsTestPasses.a lib/libMLIRAnalysis.a lib/libMLIRDialect.a lib/libMLIREDSC.a lib/libMLIROptLib.a lib/libMLIRParser.a lib/libMLIRPass.a lib/libMLIRQuantizerFxpMathConfig.a lib/libMLIRQuantizerSupport.a lib/libMLIRQuantizerTransforms.a lib/libMLIRSPIRV.a lib/libMLIRSPIRVTestPasses.a lib/libMLIRSPIRVTransforms.a lib/libMLIRTransforms.a lib/libMLIRTransformUtils.a lib/libMLIRTestDialect.a lib/libMLIRTestIR.a lib/libMLIRTestPass.a lib/libMLIRTestTransforms.a lib/libMLIRSupport.a lib/libMLIRIR.a lib/libMLIROptLib.a lib/libLLVMSupport.a lib/libLLVMCore.a lib/libLLVMAsmParser.a lib/libLLVMX86CodeGen.a lib/libLLVMNVPTXCodeGen.a lib/libMLIRAVX512.a lib/libMLIRLLVMAVX512.a lib/libMLIRROCDLIR.a lib/libMLIRSPIRVSerialization.a lib/libMLIRAffineToStandard.a lib/libMLIRAffineTransforms.a lib/libMLIRVectorToLLVM.a lib/libMLIRQuantizerFxpMathConfig.a lib/libMLIRFxpMathOps.a lib/libMLIRQuantizerSupport.a lib/libMLIRQuant.a lib/libMLIRSPIRV.a lib/libMLIRParser.a lib/libMLIRGPUtoCUDATransforms.a lib/libLLVMNVPTXCodeGen.a lib/libLLVMipo.a lib/libLLVMVectorize.a lib/libLLVMInstrumentation.a lib/libLLVMLinker.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a lib/libMLIRTargetNVVMIR.a lib/libMLIRGPU.a lib/libMLIRNVVMIR.a lib/libMLIRTargetLLVMIRModuleTranslation.a lib/libMLIRLLVMIRTransforms.a lib/libMLIRTranslation.a lib/libLLVMIRReader.a lib/libMLIRVectorToLoops.a lib/libMLIRTestDialect.a lib/libMLIRLinalgTransforms.a lib/libMLIRLinalgAnalysis.a lib/libMLIRLinalgEDSC.a lib/libMLIRLinalgUtils.a lib/libMLIRLinalgOps.a lib/libMLIRStandardToLLVM.a lib/libMLIRLLVMIR.a lib/libMLIROpenMP.a lib/libLLVMAsmParser.a lib/libLLVMFrontendOpenMP.a lib/libMLIRStandardToStandard.a lib/libMLIRDialect.a lib/libMLIRTransforms.a lib/libMLIRVector.a lib/libMLIRTransformUtils.a lib/libMLIRLoopAnalysis.a lib/libMLIRPass.a lib/libMLIRAnalysis.a lib/libMLIRAffine.a lib/libMLIRLoopOps.a lib/libMLIRStandardOps.a lib/libMLIREDSC.a lib/libMLIRLoopLikeInterface.a lib/libMLIRSideEffects.a lib/libMLIRCallInterfaces.a lib/libMLIRControlFlowInterfaces.a lib/libMLIRInferTypeOpInterface.a lib/libMLIRIR.a lib/libMLIRSupport.a lib/libLLVMAsmPrinter.a lib/libLLVMDebugInfoDWARF.a lib/libLLVMCFGuard.a lib/libLLVMGlobalISel.a lib/libLLVMSelectionDAG.a lib/libLLVMCodeGen.a lib/libLLVMScalarOpts.a lib/libLLVMAggressiveInstCombine.a lib/libLLVMInstCombine.a lib/libLLVMBitWriter.a lib/libLLVMTransformUtils.a lib/libLLVMTarget.a lib/libLLVMAnalysis.a lib/libLLVMProfileData.a lib/libLLVMObject.a lib/libLLVMMCParser.a lib/libLLVMBitReader.a lib/libLLVMCore.a lib/libLLVMRemarks.a lib/libLLVMBitstreamReader.a lib/libLLVMTextAPI.a lib/libLLVMX86Desc.a lib/libLLVMMCDisassembler.a lib/libLLVMMC.a lib/libLLVMBinaryFormat.a lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a lib/libLLVMSupport.a -lz -lrt -ldl -ltinfo -lpthread -lm lib/libLLVMDemangle.a

#2

On my machine I see 3s in release mode and 60s (!) in debug, but I also see 0.06s (and <3s in debug) when using lld instead of bfd, have you tried -DLLVM_ENABLE_LLD=ON?

Some of the the LLVM dependencies aren’t needed (others are because the LLVM dialect uses a bunch of LLVM IR internals, and the GPU conversion pass uses the NVPTX backend), but it does not matter much for the link time: the linker does not pull in any unused file from static archives.
We could cut these dependencies to reduce the overall build time, but I suspect ninja check-mlir will still build them for the other testing binaries than mlir-opt (for instance the runners which involved the JIT and so all the codegen infra anyway).

(by the way the link your posting isn’t from upstream, it included a bunch of your private libraries)

1 Like
#3

Thanks. I’m unable to use -DLLVM_ENABLE_LLD=ON out of the box - cmake complains with Host compiler does not support '-fuse-ld=lld' when running:
cmake -G Ninja ../llvm -DLLVM_ENABLE_PROJECTS=mlir -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_LLD=ON

I have clang 9.0.1. The times I posted were actually from the upstream version (have edited the link line now). Your 60s debug link time may partly be due to available memory / swapping.

#4

@bondhugula Are you sure cmake is actually finding your clang-9? it tends to look in odd places and might be finding something which is not what you expect.

#5

My CMakeCache.txt shows CMAKE_ASM_COMPILER:FILEPATH=/usr/lib64/ccache/clang and

/usr/lib64/ccache/clang --version
clang version 9.0.1 (Fedora 9.0.1-2.fc31) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/bin

#6

You need to have CMAKE_C_COMPILER and CMAKE_CXX_COMPILER also pointing to the right place.
LLVM_ENABLE_LLD is equivalent to LLVM_USE_LINKER=lld, so if your lld is named something else, you can specify the name exactly using LLVM_USE_LINKER.

when you run cmake for the first time you should see:
– The C compiler identification is Clang 8.0.1
– The CXX compiler identification is Clang 8.0.1
– The ASM compiler identification is Clang
– Found assembler: /wrk/hdstaff/stephenn/llvm-8/build/bin/clang-8
– Check for working C compiler: /wrk/hdstaff/stephenn/llvm-8/build/bin/clang-8
– Check for working C compiler: /wrk/hdstaff/stephenn/llvm-8/build/bin/clang-8 – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /group/xrlabs/tools/x86_64_RHEL7/bin/clang+±8
– Check for working CXX compiler: /group/xrlabs/tools/x86_64_RHEL7/bin/clang+±8 – works

However, once this is set in the CMakeCache.txt, then cmake doesn’t seem to go searching for it again, even if you specify -D’s on the command line. This is one of my major annoyances with cmake.

1 Like
#7

And I’ve suddenly realized my own configuration is wrong, since it’s pointing to compilers in two different directories… ARGH.

#8

Thanks @stephenneuendorffer. The issue was simpler - I didn’t have lld installed! :slight_smile: The error message The host compiler does not support -fuse-ld=lld didn’t help, but the cmake error log had:

clang-9: error: invalid linker name in argument '-fuse-ld=lld'

#9

Thanks @mehdi_amini @stephenneuendorffer. With -DLLVM_ENABLE_LLD=ON, it’s 0.45s to link mlir-opt for the release/assert build and 2.0s for the debug build.

1 Like
#10

I’m going to file an issue for the build time part. If any of those dependences are eliminated, it will really reduce mlir-opt’s build time - right now, any change to unrelated LLVM infrastructure is forcing a rebuild of all of those LLVM parts during an mlir-opt rebuild. But as @mehdi_amini points out, this may not help with check-mlir.