Runtime directory fallback

Hi,

We have discussed this topic many times online and at EuroLLVM last year, I want to try to find a solution that I can upstream.

Currently if you build compiler-rt via runtimes it will create a directory per target where the runtime libraries get stored. The problem is that the driver has a fallback to the non-per-target style, and this fallback will then report the error that gets very confusing if you misspelled the target name or have a problem with your install. Example if you have your builtin lib in the folder x86_64-unknown-linux-gnu and you pass the target x86_64-pc-linux-gnu to clang it will report that it can’t find linux/libclang_builtins_x86_64.a which is not correct and generally confusing.

We have a patch in our tree to just disable the fallback since we don’t use the other style of directories. But I want to find a solution we can upstream.

One solution is to put the fallback behind a preprocessor define that gets set by CMake depending on the value of the per_target variable.

Another solution is an opt-in preprocessor define that removes the fallback.

Or we can use a runtime flag to disable the fallback or control the “style”. But this feels a bit over-complicated.

WDYT @hansw2000 @petrhosek @mstorsjo @MaskRay

I have run into this problem many times and the error message is very confusing. Is the per-target runtime layout the default now?

I’ve run into this a few times, e.g. when I use -fsanitize=address without ninja asan first. Thanks for proposing a solution.

I think we can:

  • return getLibraryPaths()[0] in case a runtime file is not found. Some targets like AIX seem to have 2 entries in LibraryPaths.
  • Disable // Fall back to the old expected compiler-rt name when LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is defined.

The current default is:

if(CMAKE_SYSTEM_NAME MATCHES "BSD|Linux|OS390")
  set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default ON)
else()
  set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default OFF)
endif()
set(LLVM_ENABLE_PER_TARGET_RUNTIME_DIR ${LLVM_ENABLE_PER_TARGET_RUNTIME_DIR_default} CACHE BOOL
  "Enable per-target runtimes directory")

I would prefer the second option, but I would be OK with the first too.

I also prefer option 2.

I think this option is also on when you build with LLVM_ENABLE_RUNTIMES so it should be most builds these days.

The two bullet points are combined (I think we can report any entry in LibraryPaths). See [Driver] Improve error when a compiler-rt library is not found by MaskRay · Pull Request #81037 · llvm/llvm-project · GitHub for the idea.

I would prefer a different solution.

Overall, I dislike cases where a Clang binary has this kind of decision hardcoded into it. Yes, we do have a precedent for such things with CMake options like CLANG_DEFAULT_LINKER=lld (and yes, I do use such configurations myself occasionally), but they generally hinder use of Clang, when it e.g. can be used for other targets than the default one it was originally configured for.

Runtime configurability (which can be controlled with a default config file - which can be overridden with a --no-default-config if using the clang binary for a different target) generally is preferrable to me. But for this case, I’m not sure if it is warranted to add yet another command line option to control this detail.

As for the suggested patch; I would prefer to not bake the choice of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR into something visible in llvm-config.h. (And if we really do go that way, I think config.h which is private to the LLVM build, would be more applicable? But perhaps that doesn’t work correctly if LLVM and Clang are built in two separate build steps?)

To me, runtimes is something that not necessarily is built at the same time as Clang (I never do), so the fact that runtimes can be built at the same time is mostly a convenience.

Overall, I would prefer if we don’t stop looking for the old arch-suffixed form of libraries, even if the per-target runtime directory form might be the preferred one.

If the main issue is the error message is hard to reason about (which I certainly would agree with), I think we could achieve it by probing for the old filename, and using the per-target runtime directory as fallback (which gets printed in the error message).

We could achieve that with a change like this:

diff --git a/clang/lib/Driver/ToolChain.cpp b/clang/lib/Driver/ToolChain.cpp
index 388030592b48..fe47d2d85092 100644
--- a/clang/lib/Driver/ToolChain.cpp
+++ b/clang/lib/Driver/ToolChain.cpp
@@ -655,20 +655,28 @@ std::string ToolChain::getCompilerRT(const ArgList &Args, StringRef Component,
   // Check for runtime files in the new layout without the architecture first.
   std::string CRTBasename =
       buildCompilerRTBasename(Args, Component, Type, /*AddArch=*/false);
+  SmallString<128> Path;
   for (const auto &LibPath : getLibraryPaths()) {
-    SmallString<128> P(LibPath);
-    llvm::sys::path::append(P, CRTBasename);
-    if (getVFS().exists(P))
-      return std::string(P);
+    Path = LibPath;
+    llvm::sys::path::append(Path, CRTBasename);
+    if (getVFS().exists(Path))
+      return std::string(Path);
   }
 
-  // Fall back to the old expected compiler-rt name if the new one does not
-  // exist.
+  std::string FallbackPath = std::string(Path);
+
+  // Check the old expected compiler-rt name if the new one does not exist.
   CRTBasename =
       buildCompilerRTBasename(Args, Component, Type, /*AddArch=*/true);
-  SmallString<128> Path(getCompilerRTPath());
+  Path = getCompilerRTPath();
   llvm::sys::path::append(Path, CRTBasename);
-  return std::string(Path);
+  if (getVFS().exists(Path))
+    return std::string(Path);
+
+  // If none is found, use a file name from the new layout, which may get
+  // printed in an error message, aiding users in knowing what Clang is
+  // looking for.
+  return FallbackPath;
 }
 

(Untested, only compile tested.)

Overall, this issue feels more generally like something that I do feel appears in a number of places when trying to set up a toolchain with headers/libraries - Clang looks for things in a bunch of different places, but it’s not obvious which ones were considered but skipped. (Some paths do get added even if they don’t exist, so they appear in e.g. clang -v, listed as discarded as nonexistent, but many paths are just probed and skipped.) For that, I wonder if it would be good to have an option to Clang driver to request it to verbosely mention all the paths that it tests/evaluates. E.g. for headers, there’s lots of different combinations of sysroot, /usr/include, triples (some in the clang-normalized form, some in <arch>-linux-gnu form for debian multiarch) that do get tested.

Clang has a version lock with the resource directory (which contains the compiler-rt library files), so using a clang with -resource-dir= pointing to a different installation may not work.

But perhaps that doesn’t work correctly if LLVM and Clang are built in two separate build steps?)

I agree that changing llvm/include/llvm/Config/llvm-config.h.cmake is not pretty. Clang doesn’t include llvm/Config/llvm.h. I vaguely remember it is due to the stand-alone build requirement.

We could achieve that with a change like this:

Thanks for the suggestion. We can do this to avoid a llvm-config.h change. We don’t want external users to read LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.

Since I’ve prepared the change, I remember that I made a similar change few years ago but gave up probably because a few driver tests needed (and need) fixing… I think clang/test/Driver tests need some cleanup before we can improve ToolChain::getCompilerRT… Android cleanup [Driver] Remove "-android" from compiler-rt library names for legacy LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=off by MaskRay · Pull Request #81044 · llvm/llvm-project · GitHub

Overall, this issue feels more generally like something that I do feel appears in a number of places when trying to set up a toolchain with headers/libraries - Clang looks for things in a bunch of different places, but it’s not obvious which ones were considered but skipped. (Some paths do get added even if they don’t exist, so they appear in e.g. clang -v , listed as discarded as nonexistent, but many paths are just probed and skipped.) For that, I wonder if it would be good to have an option to Clang driver to request it to verbosely mention all the paths that it tests/evaluates. E.g. for headers, there’s lots of different combinations of sysroot, /usr/include, triples (some in the clang-normalized form, some in <arch>-linux-gnu form for debian multiarch) that do get tested.

On Linux, one can invoke strace -e file /tmp/Debug/bin/clang a.c '-###' to list the poked files. The list is very long… I think listing the poked files will be very noisy and the change could be very intrusive to clang/lib/Driver

Ok, nice to hear that you agree that changing llvm-config.h isn’t pretty.

I presume you mean llvm/Config/config.h? Yes, that’s quite plausible that it’s due to allowing standalone builds.

Out of different alternatives, I maybe even would prefer having it as a separately controllable option, like CLANG_DEFAULT_LINKER, rather than implicitly building in a behaviour change from a runtime specific option.

Thank you for considering my suggestion!

Oh, I’ll try to keep that in mind next time I need to do such debugging.

Yes, I guess the list can be quite long. I envisioned something like changing/instrumenting all calls like getVFS().exists() - or perhaps only some of them, in order to get a high level overview of which combinations are considered and what is decided. What I had in mind could potentially also could be done piecemeal (for the common files and some toolchains first, then others later), although perhaps it’s not good to do such changes incompletely?

Anyway, that was just a stray idea that came to mind.

While the error message is one part of it, we had an issue where the fallback ended up loading the wrong runtime. I don’t remember the details 100% but due to some flags that where set in our build system, clang fell back on the builtin libraries from MSVC instead of using the ones bundled. This is when we wrote this patch specifically, both because the error message made it harder to debug and to avoid the driver to fallback and load something we didn’t expect.

So while changing the order would be good for my use case, I wonder if it could lead to the inverse effect for toolchains where per_target_dir is not used?

So my vote will still be on a way to remove the fallback.

Hmm, right, I see. (In most cases, Clang refers to these libraries by passing an absolute path, but I guess there are some cases where that’s not done.)

Yeah, certainly - we’d be shuffling the issue around mostly. Hopefully into a place where the error is less misleading, but I guess that’s hard to say for sure.

Right - I don’t mind adding e.g. a CMake option for making it skip the fallback, then. However I would kind of prefer for it to not default to true.

In my setup, I first build llvm+clang in one build. I don’t touch LLVM_ENABLE_PER_TARGET_RUNTIME_DIR there, since I’m not building any runtimes. Then in a later step I build the runtimes with separate cmake invocations pointing at llvm-project/runtimes.

Unfortunately, if we don’t make it default to true, people will in general still get error messages pointing to the old name style - unless we also go forward with the change I suggested, to use the new style path as fallback.

Separately - if the reason for avoiding the old name is to avoid accidentally using MSVC shipped libraries - what will you do if MSVC updates and adopts the new naming style for the libraries they ship? Then we’d be back at square one. So in one sense, the distinction between old and new naming style, and avoiding libraries from another vendor within the same search path, are two slightly separate issues.

I think I favor option 2, of disabling the fallback.

Do we have line-of-sight to removing support for LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=OFF? Would Martin’s problems go away if we did that, since there is no possibility of mismatch, as long as one is using matched versions of clang and compiler-rt? If we eliminate this option altogether, then there is no chance for mismatch.

I seem to recall that not everyone or all platforms were able to migrate to the new library directory layout. If we have a path to finishing this migration and eliminating the fallback that way, that would be a great simplification.

Exactly - not all platforms can migrate, for all aspects of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.

For my targets, I could switch, but I find the old approach more robust (less risk for subtle triple differences etc), so I haven’t changed, yet. (I do test that setup occasionally though.)

The concept of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR consists of two kinda separate things actually.

For compiler-rt, essentially all accesses within Clang is in common shared code. So I think there’s no target specific variability for that - so that should work for all targets, provided that build and use always are done with the exact same spelling of the target triple.

For other runtimes, in particular libc++, code for detecting and adding such paths is scattered all over the target specific files in Clang. I remember that a handful of targets had been updated to check in the per-target runtime directory, but not necessarily all of them, at least last time I checked. So not all targets might be ready to switch over, for these runtimes.

Finally, it’s a matter of taste. For the compiler-rt libraries, installed in <root>/lib/clang/<version>/lib, there’s not much of a difference between having windows/libclang_rt.builtins-x86_64.a vs x86_64-w64-windows-gnu/libclang_rt.builtins.a - I don’t mind that much (I just postpone switching unless I have to).

However for other libraries, like libc++, my current setup is cleaner, IMO.

My toolchain distribution is set up for cross compilation. Currently, under the toolchain root, I have subdirectories for each cross target, containing the subdirectories bin, include and lib. Within these, I have all the files that are specific for this target, all libraries (import libraries and static libraries etc). So I have libc++ in <root>/x86_64-w64-mingw32/lib/libc++.a, next to <root>/x86_64-w64-mingw32/lib/libmingwex.a and all other files that are specific for that target.

If I were to switch this over to the per-target runtime layout, the few specific libraries that are installed this way, would need to be in <root>/lib/x86_64-w64-windows-gnu instead - segregating the target specific files; some in <root>/x86_64-w64-mingw32/lib while others are in <root>/lib/x86_64-w64-windows-gnu. And the runtime DLLs (that only are needed at runtime, not during linking), I would still install in <root>/x86_64-w64-mingw32/bin.

For toolchains that don’t concern themselves with cross compilation, having the libraries in <root>/lib/<triple> clearly is a massive improvement over having them in <root>/lib though, I definitely agree with that.


So to recap, I wouldn’t mind much (only a little, but I can take it :wink: ) if we’d make the per-runtime target dir layout mandatory for compiler-rt. (But IIRC Apple targets have their own, different way of handling that anyway.) But I kinda don’t want to hard require LLVM_ENABLE_PER_TARGET_RUNTIME_DIR for other runtimes. I can make that work too, if that’s necessary, but I would prefer not to.

Do we have a consensus on this issue yet? It seems like most are in favour of being able to disable the fallback. But some disagreement on how to remove the fallback? What’s the preferred solution in this case if we want to avoid the config header?

Yes, no objection to that.

I’m not entirely sure.

IMO, it’s not so much about “avoid the config header” - the main issue, is that we can’t generally connect LLVM_ENABLE_PER_TARGET_RUNTIME_DIR, from the time of building Clang, to the layout of the runtimes it will be used with later, as they don’t need to be built in the same step.

Do the others, @MaskRay, @rnk, agree about that assessment of the situation?

The most robust, least conflicting solution would to be add a new config option, à la the existing CLANG_DEFAULT_RTLIB and similar. I’m overall not a fan of them, not a fan of adding more of them, but they’re probably the least disruptive way forward here.

Unfortunately, as follows from the earlier conclusion that we can’t connect this to the build-time LLVM_ENABLE_PER_TARGET_RUNTIME_DIR setting, we can’t really enable this new CLANG_DEFAULT_DISABLE_CLANGRT_FALLBACK (not the best name, better are welcome) option by default either. And since it’s not enabled by default in the majority of cases, we won’t get the benefit of less confusing error messages by default either. And that’s sad, as it would apparently be quite beneficial. :frowning:

I managed to clean up the tests, and [Driver] Improve error when a compiler-rt library is not found by MaskRay · Pull Request #81037 · llvm/llvm-project · GitHub should address the original report.

Since there are both LLVM_ENABLE_PER_TARGET_RUNTIME_DIR on and off operating systems.
Technically the overhead supporting off runtime directories using on clang isn’t that large.

[Driver] Improve error when a compiler-rt library is not found by MaskRay · Pull Request #81037 · llvm/llvm-project · GitHub has landed and should improve the error message as LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on configurations become more prevailing.

CC some folks who may help AIX migrate to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=on and remove an AIX special case:

@@ -681,19 +681,29 @@ std::string ToolChain::getCompilerRT(const ArgList &Args, StringRef Component,
   // Check for runtime files in the new layout without the architecture first.
   std::string CRTBasename =
       buildCompilerRTBasename(Args, Component, Type, /*AddArch=*/false);
+  SmallString<128> Path;
   for (const auto &LibPath : getLibraryPaths()) {
     SmallString<128> P(LibPath);
     llvm::sys::path::append(P, CRTBasename);
     if (getVFS().exists(P))
       return std::string(P);
+    if (Path.empty())
+      Path = P;
   }
+  if (getTriple().isOSAIX())    ///// special case, should be removed
+    Path.clear();

@amy-kwan @qiongsiwu @qcf

See [Driver] Restore arch suffix for PS and Windows by pogo59 · Pull Request #89775 · llvm/llvm-project (github.com) because we were seeing link failures with UBSan on Windows. MSVC 2019 and 2022 both distribute libraries with the arch suffix; so does PlayStation although there’s really not much justification for the latter.