TL;DR: We should make LLVM_LINK_LLVM_DYLIB (and potentially later CLANG_LINK_CLANG_DYLIB) default to ON in CMake for non-Windows platforms. While all tradeoffs have pros and cons, I think it’s clear at this point that the benefits outweigh the negatives. The DSO build makes building LLVM more accessible to beginners, and I believe that’s important to the long-term health of the project, so we should make it the default.
These are the benefits of using the libLLVM DSO:
- The LLVM DSO uses much less diskspace (18GiB vs 3.1GiB for Linux Rel+asserts). This multiplies by ~5x if you enable debug info.
- Linking the LLVM tools with the DSO is faster and uses less memory. Linking the tools is a major bottleneck for the
check-llvm/clang/mlirtesting workflow. Running multiple high-memory link steps inhibits development on entry-level hardware, which beginners tend to have. - Using the LLVM DSO by default aligns our default build config with Debian, Fedora, and other Linux distro build configs, which all use libLLVM*.so. If you are a vendor who cares about package size, this enables creating a smaller, more modular collection of toolchain packages to distribute.
The downsides of the libLLVM DSO are:
- Minor hit to startup time on ELF platforms. Recent data says this is marginal, but I’ll expand on this below, since the received wisdom is that this is a major blocker. Non-ELF platforms generally have faster loaders, or collude with the kernel to share relocated pages between processes.
- The DSO increases the number of compile actions required to rebuild a single tool binary, slowing down iteration times when working with small test inputs. For example, if I make a change to IR/ headers, and I can test my change with
llvm-as/disoropt, the static build would allow me to test my changes without building the entire LLVM DSO, which includes llvm/lib/Target. Static linking, or fine-grained shared linking (BUILD_SHARED_LIBS=ON), can offer faster iteration times in cases like this. - The DSO build doesn’t validate internal library dependencies. In the static build, static links often fail if you forget a library dependency. This helps us maintain boundaries between layers in LLVM’s internal architecture. Alternative, unsupported build systems such as Bazel require accurate dependency information. This gap can also be covered with continuous BUILD_SHARED_LIBS testing, which I believe we already have on buildbot, because I broke it at some point.
- Tool binaries are less hermetic. They already search for paths relative to the executable path, but if you add shared library dependencies, that’s one more path to manage if you want to copy a tool binary around. IIRC we do this in a regression test, which IMO is questionable.
- The DSO limits whole-program visibility in LTO-enabled builds. This is a major consideration for a vendor attempting to build a highly optimized toolchain binary, but is probably not a major consideration for our default build configuration.
Regarding the startup time overhead, this came up on LKML in 2021, and it resulted in this widely circulated post from Linus Torvalds “shared libraries are not a good thing in general”. Regardless, all the major Linux distros (Fedora, Debian (see llvm-toolchain package rules)) still use this configuration, presumably because it allows them to split up the toolchain into smaller, more modular packages (clang, llvm, lld, the rest), which use far less disk space.
However, recent benchmark data from @mstorsjo shows that this startup time hit is low. For short-lived executions like clang --version, the mean wall time goes from 9.5ms to 12.5ms, resulting in a 3ms overhead or +31%. However, if you add 3ms to a 1sec compilation action, we’re talking about a 0.3% perf hit. The overhead is lost in the noise on large compile actions, such as the ~20s sqlite3.c aggregate compilation tested in that benchmark.
I think part of why it is received wisdom that dynamically linked LLVM builds are slow comes from experience running the LLVM test suite, where the dynamic linker often has been the bottleneck, because our test processes are short-lived and process startup is the bottleneck. I did a quick A/B comparison of check-llvm, and my numbers show that the DSO build is faster, but I don’t trust my configuration and this deserves external validation.
I think 3ms of startup overhead is something worth optimizing, but this seems like a tradeoff that a vendor should consider, not something people doing their first LLVM build should have to think about. I believe there is also additional room to explore optimization flags such as -Bsymbolic-funcitons-non-weak, -fvisibility-inlines-hidden, as well as finishing up the LLVM_ABI annotations so we can build with -fvisibility=hidden. We can also continue optimizing dynamic relocations, as @chandlerc recently did in a series of PRs improving Clang’s builtin string tables a few months back. These are all good next steps, but I think the pros already outweigh the cons, so they don’t need to be prerequisites.
Regarding platform support, LLVM_LINK_LLVM_DYLIB is not supported on Windows. Once we have accurate LLVM_ABI annotations (I hope this happens), then we can re-evaluate turning this on for Windows.
Simply put, static linking is a bad default for LLVM today. The build directories are too big, and the many redundant static links use a lot of memory. Flags to optimize the linker are already explicitly mentioned in our getting started guide to work around these large static link steps. New developers regularly complain about build directory size (1), and most recently, I personally had a 111GB build directory that caused me to explore merging clang unit tests.
In order to make the project more accessible, we really should adjust our defaults so that building LLVM doesn’t require so many resources. Using large shared libraries by default would be a good step in that direction.
cc relevant folks @mstorsjo @arsenm @nikic @tstellar @MaskRay @compnerd