Improve autolinking of compiler-rt and libc++ on windows with lld-link


At EuroLLVM, I had some discussions with @mstorsjo, @hansw2000, @wanders, and @petrhosek about how to improve the out-of-the-box behavior for lld to find compiler-rt builtins, sanitizers, and libc++. Later this discussion included @rnk in an email chain.

I will summarize the current situation and suggest steps to move forward.


Some background for people that might not be that familiar with Windows. When linking to compiler-rt/libc++ on Linux, you usually invoke clang as the linker, and the clang driver will construct the linker line to include the path to compiler-rt/libc++ and add the library on the link line.

On Windows, the norm is to invoke the compiler and linker separately. The compiler can add dependent libraries into the .obj files with the help of pragmas. But the linker needs to find the path to compiler-rt/libc++ itself. MSVC solves this by opening a “developer terminal”, which will populate the shell with some environment variables that point the linker to the correct directories.

The problem

We need to use the compiler-rt builtins for code to work in several cases. This includes int128 support. Currently, the following is a problem:

int main(int argc, char **argv) {
        __int128 a = 123;
        __int128 b = 1;
        return a / b;

C:\src\temp>clang-cl /c

C:\src\temp>lld-link a.obj
lld-link: error: undefined symbol: __divti3
>>> referenced by a.obj:(main)

The same can be said for sanitizers and libc++. We can’t use the clang driver to construct the linker line, so things like: clang-cl -fsanitize=address /c && lld-link a.obj

Each runtime has different problems. Here are some of the issues we are trying to solve:

  • Find the path (libpath) to the libraries
  • Find and insert the library name into the obj files.
  • Sanitizers (specifically ASAN) have a complex matrix of names and libraries that must be linked in different configurations. MSVC tries to solve this with the new /inferasanlibs (/INFERASANLIBS (Use inferred sanitizer libs) | Microsoft Learn)

The “per-target” problem

Finding and constructing the path to the compiler-rt where LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is set to false is much easier than when it’s set to true.

When this option is set to false, the path to the compiler-rt’s are: lib/clang/16/lib/windows, and the architecture is encoded in the filename (clang_rt.builtins-x86_64.lib). Constructing this is easy since all that information is available within the linker context.

But when this option is set to true, the path looks like this lib/clang/16/lib/x86_64-pc-windows-msvc. Which contains the triple; lld doesn’t know about the triple, and constructing it from scratch might be tricky since we might not have all the relevant information.


Since there is a lot of information that is not available to lld, we can either:

  • Teach LLD more stuff, like how to create the triple. I made an attempt here ⚙ D151188 [LLD][COFF] Add LLVM toolchain library paths by default. in the llvm::Triple LinkerDriver::constructTriple() method, but it becomes a lot of assumptions and messy code.
  • Make decisions in the clang driver and transfer them via --dependent-lib to the linker.

I think solution two is probably best, and it would look something like this:

  • clang driver will insert the builtin library relative path into the obj file with the prefixed target directory: i.e clang --depentent-lib x86_64-pc-windows-msvc/compiler-rt.builtins.lib or windows/compiler-rt.builtins-x86_64.lib depending on the configuration of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.
  • lld will add the directory’s root to its default search paths (lib/clang/16/lib in this case).

This will work fine for builtins and ubsan.

ASAN is more complicated since the support matrix will need to know if we are building a DLL or executable and which CRT we are linking to. To retain compatibility with MSVC, I think the best would be to implement /inferasanlibs in LLD, but to avoid the target resolution in LLD as discussed above, we probably need clang to insert some placeholder as the dependent-lib, for example, x86_64-pc-windows-msvc/<asanlib>. Not quite sure about this yet.

Finally libc++. Currently, libc++ inserts pragmas/dependent-libs to have the linker link to them. This will work with LLVM_ENABLE_PRE_TARGET_RUNTIME_DIR=OFF but not when it’s ON. In which case, it makes things a bit more complicated since we would have “rewrite” the dependent-lib line and add the target triple into it… to move triple awareness to lld.


  • Windows build systems invoke the linker directly instead of clang to link.
  • We need to improve how LLD finds libraries on Windows platforms to support builtins/sanitizers/libc++ better out of the box.
  • This is complex because of the LLVM_ENABLE_PER_TARGET_RUNTIME_DIR setting since those directories contain the triple, and LLD doesn’t know about the triple
  • It would be nice to make these decisions in the clang driver and transfer that information to the linker by encoding it in the .obj files.
  • There are caveats and edge cases.

Hope to get the community feedback on this topic and suggestions on how to move forward since all ways forward seem to contain some drawbacks.

I have^TM no stakes in Windows anymore. You could ship a lld-link.bat, which is a clang in disguise.

I think you wanted to say that you can ship a lld-link.bat and not cannot.

We did consider this, but it would be very complicated bat script or changes to the clang driver in that case, considering how different the link.exe command line interface is from what clang is today. It’s possible, but it’s not a small effort.

Sorry. Typos. Indeed the interface would be completely different.

I’d suggest a small tweak to this; for the case when not using the per-target runtime directory, we should just embed the clang_rt.builtins-x86_64.lib name into the object file, and lld should look both in lib/clang/16/lib and lib/clang/16/lib/windows. As MSVC itself ships some variants of clang_rt.builtins-*.lib in their own library directories (brought in mostly by coincidence I think, as they build compiler-rt for ASAN), that setup falls back transparently on the MSVC provided builtins if the clang build doesn’t happen to have one for the right configuration.

Secondly; clang itself doesn’t know if the runtimes are built with LLVM_ENABLE_PER_TARGET_RUNTIME_DIR, but it needs to probe a bit and see if it finds them in one path or the other, before deciding what to emit into the object files.

I suppose clang could also look at the cmake flag, but yes probing sounds better. It does mean that those files need to be available for the compiler to find when running in a distributed build setting, but I think that’s okay.

We discussed making LLVM_ENABLE_PER_TARGET_RUNTIME_DIR a compile-time configuration and removing the fallback since it creates confusion. @petrhosek

It can’t look at it directly; remember that the runtimes can be built in an entirely separate cmake step by pointing cmake at llvm-project/runtimes. If we go down the route of making it a hardcoded setting, it’d probably be more like the existing flags for hardcoding defaults like CLANG_DEFAULT_CXX_STDLIB or similar. (All of those flags for hardcoding defaults into Clang are a bit problematic wrt cross compilation though; you might want to have one hardcoded default for a cross target, but a different one when using the same Clang executable for building things natively.)

Probing the type/style of builtins that are available and how to link them, is already how LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is handled. The difference here would be that the same probing would have to happen while compiling object files and deciding what linking variant to embed into them, not only at link time (which usually doesn’t pass through Clang at all for msvc style configs).

Are we working to make LLVM_ENABLE_PER_TARGET_RUNTIME_DIR true everywhere by default? If yes, then let’s focus on making things convenient in that configuration, and not worry too much about the usability of the old configuration, or mismatches between separate runtime builds for different targets using different configurations. Users always have the escape hatch of adding -libpath: flags if they are using the old config.

I also wouldn’t try to structure things to rely on the MSVC installation.

I think the downside of approach 2 is that I don’t think it is compatible with link.exe, but it needs testing. My recollection is that -defaultlib: directives with relative paths are not handled in a sensible way. Perhaps link.exe only searches for relative paths starting from the working directory, I can’t recall, it needs confirmation.

That leads me back to approach 1, which I guess is to teach lld-link to guess the triple. While this is somewhat fragile, we can pretty much assume that lld-link is targetting a windows-msvc environment. For mingw, users will use the other driver, and typically call through the clang driver anyway. The only unknown is really the architecture, and at this point, there are just a few of those. We get ourselves into trouble with all the microarch variants of them, however: i386, i686, x86_64h, thumbv7, etc.

Another way forward would be to invent some way to communicate the triple to lld-link through the object file, like a custom directive flag, or some other custom section like .llvm_addrsig. That seems like a last resort.

So in conclusion, I think I’m back where I was in the code review thread here: ⚙ D151188 [LLD][COFF] Add LLVM toolchain library paths by default. We should go forward with the triple guessing, and not worry about microarchitectures or other custom triple spelling adjustments. People doing that can pass -libpath: flags.

I tested that in ⚙ D151188 [LLD][COFF] Add LLVM toolchain library paths by default. and it seemed to work pretty well:

C:\src\temp>clang-cl /c -Xclang --dependent-lib=windows\clang_rt.builtins-x86_64.lib && link a.obj "/libpath:C:\Program Files\llvm\lib\clang\16\lib"
Microsoft (R) Incremental Linker Version 14.29.30145.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Well, I don’t see any issues with approach 2, then, other than the one @mstorsjo outlined relating to libc++ pragmas.

Are we OK with Clang special casing #pragma comment(lib, "c++.lib") to prefix it with “${triple}/”? That could break things if libc++ is provided in some other library search path controlled by the user.

I’m concerned about the case where someone builds a static library, then links it with MSVC link.exe. (The link step may happen on a different machine, so the compiler doesn’t know where clang will be installed at link-time.) Currently, that basically just works if you use MSVC STL and don’t need anything from builtins. We should document the expected interaction.

If the interaction is “the user must pass some -libpath argument to the linker”, I guess that’s okay.

Note that for 128-bit divide in particular, there’s a fallback path that avoids using compiler-rt.builtins; we just need to fix the call to setMaxDivRemBitWidthSupported() in X86ISelLowering.cpp. That might be worth doing independent of whatever we decide to do here.

Special casing doesn’t seem great, but could Clang expose the triple via a preprocessor macro so the pragma could include the per-target runtime dir?

If the interaction is “the user must pass some -libpath argument to the linker”, I guess that’s okay.

Yeah, I think that seems fair, especially if we document it in a way that’s easily found when searching based on what the typical error would look like.

I’m following this thread with interest, because it seems directly related to the problems I’m having with flang 16 on windows, which – even for a trivial hello-world example – wants to link __divti3 and thus needs compiler-rt.builtins (and then fails with something that looks like /MD vs. /MT; details)

Would be great, especially if this could benefit the flang situation as well!