RFC: Place libs in Clang-dedicated directories (affects openmp, libcxx, libunwind, compiler-rt)

Hi Petr,

I’ve implemented https://reviews.llvm.org/D59013 which moves libunwind, libc++abi and libc++ output to lib/ and include/ in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR build, feedback is welcome.

Thanks for working on that. Sorry, I was traveling and didn’t have a
chance to take a look before you pushed.

So, if LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True, then all libraries in
lib/x86_64-unknown-linux-gnu are now early in my library search path.
Is that right?

One of the objections to previous patches I’ve seen is that promoting
non-Clang-dedicated directories (that is, their names don’t contain
“/clang/”) is not safe for backward compatibility because that can
affect library search order for system libraries not installed by
Clang. Doesn’t your patch have that issue?

I believe so. Would the alternative be inserting a clang/ component, i.e. lib/clang/x86_64-linux-gnu?

It sounds sufficient to me. Just to have something to compare to, there was also an example layout here:

https://reviews.llvm.org/D30733#697781

I’m not sure whether that’s any better.

Personally, I prefer using triples rather than nested directories. It’s what Clang already uses setting the target and it’s also used by other toolchains and systems.

I could make that change, my patch has been reverted because it broke Windows bots so I can address this in the reland.

Another alternative would be to avoid relying on -L for libraries like libunwind, libc++abi and libc++ which will always have the issue of being a subject to collisions and instead teach the driver to use a full path just like it does for compiler-rt runtimes, but that’s a dramatic change.

What’s the advantage over a Clang- dedicated directory that’s early in the search path?

AFAIK the motivation for using full paths for compiler-rt runtimes compared to e.g. -lgcc_s was https://en.wikipedia.org/wiki/XcodeGhost which relied on injecting -L to make linker pick up different library than the one shipped with Xcode.

Hi Petr,

I’ve implemented https://reviews.llvm.org/D59013 which moves libunwind, libc++abi and libc++ output to lib/ and include/ in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR build, feedback is welcome.

Thanks for working on that. Sorry, I was traveling and didn’t have a
chance to take a look before you pushed.

So, if LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True, then all libraries in
lib/x86_64-unknown-linux-gnu are now early in my library search path.
Is that right?

One of the objections to previous patches I’ve seen is that promoting
non-Clang-dedicated directories (that is, their names don’t contain
“/clang/”) is not safe for backward compatibility because that can
affect library search order for system libraries not installed by
Clang. Doesn’t your patch have that issue?

I believe so. Would the alternative be inserting a clang/ component, i.e. lib/clang/x86_64-linux-gnu?

It sounds sufficient to me. Just to have something to compare to, there was also an example layout here:

https://reviews.llvm.org/D30733#697781

I’m not sure whether that’s any better.

Personally, I prefer using triples rather than nested directories. It’s what Clang already uses setting the target and it’s also used by other toolchains and systems.

I could make that change, my patch has been reverted because it broke Windows bots so I can address this in the reland.

Another alternative would be to avoid relying on -L for libraries like libunwind, libc++abi and libc++ which will always have the issue of being a subject to collisions and instead teach the driver to use a full path just like it does for compiler-rt runtimes, but that’s a dramatic change.

What’s the advantage over a Clang- dedicated directory that’s early in the search path?

AFAIK the motivation for using full paths for compiler-rt runtimes compared to e.g. -lgcc_s was https://en.wikipedia.org/wiki/XcodeGhost which relied on injecting -L to make linker pick up different library than the one shipped with Xcode.

Interesting. Can -resource-dir be injected in the same manner to override compiler-rt?

Do users ever depend on being able to specify -L to override Clang’s libc++, etc.?

Thanks.

Joel

My alternative to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is the preceding
bullets. In other words, you wouldn’t need to specify
LLVM_ENABLE_PER_TARGET_RUNTIME_DIR because it would effectively be
always on (except the directories might be different than now if the
version locking issue is important, as noted above). Is that what
you’re asking?

That would be my preference. I always hoped that LLVM_ENABLE_PER_TARGET_RUNTIME_DIR would eventually become the default. It would be nice to finish the Darwin support so we can completely deprecate the old layout, but I don’t know how far along beanz is in his effort. We should also update openmp to stop using the custom Android-specific runtime layout.

There’s also the unresolved question of where should libc++ headers and libraries go. Currently, in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR we use the resource dir, but some people expressed the opinion that we shouldn’t be using these for libc++ et al. since they’re not version-locked to Clang. This is different from what GCC does (e.g. GCC would use $prefix/lib/gcc/x86_64-linux-gnu/8/libstdc++.a) and it’s one of the reasons why I used the resource dir for libc++ et al. when implementing LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.

So concretely, today LLVM_ENABLE_PER_TARGET_RUNTIME_DIR uses the following layout:

headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext

The alternative that doesn’t use resource dir for libc++ would be the following:

compiler-rt:
headers: $prefix/lib/clang/$version/include
libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext

libc++, libc++abi, libunwind:
headers: $prefix/include/c++/v1
libraries: $prefix/lib/$triple/$name.$ext

Petr,

One thing I was considering tinkering with was producing additional, msan-enabled libs for libc++/libc++abi in test-release.sh and changing Clang’s driver to look for those first when “-fsanitize=memory” and “GetCXXStdlibType(Args) == ToolChain::CST_Libcxx”. This should give folks who want to use MSan a much easier path to do so. Does this make that design change easier? Say, if libc++/abi were found by looking in “$prefix/lib/$triple/$name-msan.$ext” ?

This is something that’s already supported and we use it in our toolchain. Specifically, runtimes build allows building sanitized version of runtimes, the artifacts end up in $prefix/lib/$triple/$sanitizer/$name.$ext. We use it to build ASan-instrumented versions of libc++, libc++abi and libunwind which we ship with our toolchain.

There’s currently no support in the driver since sanitizers shouldn’t affect the ABI, so it’s fine to link against the uninstrumented version and then load the instrumented one at runtime.

Hi Joel,

Sorry for the late response, was on vacation. Find a few more clarifications inline.
Hopefully still useful, even though the actual changes have already landed.

Hi Ilya,

Hi Joel,

clangd, clang-tidy and other tools do not require to be built from the same revision as the host compiler that the project uses to build code. In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
However, the internal clang headers (the ones -resource-dir points at) must correspond to the same version of the code that the clang frontend is built from.
So the aforementioned tools ship their own version of clang’s internal headers and pass -resource-dir to the clang frontend to make sure the frontend picks them up. I.e. if the host compiler is also clang, the tools must not pick the host clang’s internal headers.
The tools take other compilation arguments from a compilation database (compile_commands.json).

Thanks. That clears up a lot for me.

Naively, I would’ve thought any project (clangd, clang-tidy, or any
project external to LLVM) would specify -I to tell the compiler
(clang, gcc, or MSVC) where to find the project’s required headers
when building the project. Using -resource-dir sounds like a special
shortcut for the case where the project is clang-based (clangd or
clang-tidy) and the compiler building the project is clang. Is that
right? What do clangd and clang-tidy specify to the compiler if the
compiler is instead gcc or MSVC? Do those have an option equivalent
to -resource-dir?

The underlying “compiler” for clangd and clang-tidy is always clang (as they are calling into the clang frontend), so the tools
would just override -resource-dir as normal. This only works because clang is good enough at understanding gcc flags
and clang-cl is good enough at understanding MVSC flags (although I haven’t really checked MSVC flags work, but in
principle they could).

I don’t mean to be arguing against the usage of -resource-dir for this
purpose. I’m just trying to understand it.

Note that the internal headers is the only thing that the tools need to override, e.g. this should not affect the C++ standard library found by the tools.
For that to work, we have to make sure the internal headers is the only thing affected by “-resource-dir”, we don’t want the tools to see a different standard library (or not find a standard library at all).

Makes sense.

So far in cases where -resource-dir was used for finding libc++, replacing it with “compiler install dir” (“Driver.InstalledDir”) seemed to do the job.

Does “resource-dir used for finding libc++” imply libc++ is in the
resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?

The same recipe seemed to work in that case.
LLVM_ENABLE_PER_TARGET_RUNTIME_DIR relied on -resource-dir before D59013 landed, and seems to use install dir now.

Hi Joel,

Sorry for the late response, was on vacation. Find a few more clarifications inline.
Hopefully still useful, even though the actual changes have already landed.

Hi Ilya,

Hi Joel,

clangd, clang-tidy and other tools do not require to be built from the same revision as the host compiler that the project uses to build code. In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
However, the internal clang headers (the ones -resource-dir points at) must correspond to the same version of the code that the clang frontend is built from.
So the aforementioned tools ship their own version of clang’s internal headers and pass -resource-dir to the clang frontend to make sure the frontend picks them up. I.e. if the host compiler is also clang, the tools must not pick the host clang’s internal headers.
The tools take other compilation arguments from a compilation database (compile_commands.json).

Thanks. That clears up a lot for me.

Naively, I would’ve thought any project (clangd, clang-tidy, or any
project external to LLVM) would specify -I to tell the compiler
(clang, gcc, or MSVC) where to find the project’s required headers
when building the project. Using -resource-dir sounds like a special
shortcut for the case where the project is clang-based (clangd or
clang-tidy) and the compiler building the project is clang. Is that
right? What do clangd and clang-tidy specify to the compiler if the
compiler is instead gcc or MSVC? Do those have an option equivalent
to -resource-dir?

The underlying “compiler” for clangd and clang-tidy is always clang (as they are calling into the clang frontend), so the tools
would just override -resource-dir as normal. This only works because clang is good enough at understanding gcc flags
and clang-cl is good enough at understanding MVSC flags (although I haven’t really checked MSVC flags work, but in
principle they could).

I don’t mean to be arguing against the usage of -resource-dir for this
purpose. I’m just trying to understand it.

Note that the internal headers is the only thing that the tools need to override, e.g. this should not affect the C++ standard library found by the tools.
For that to work, we have to make sure the internal headers is the only thing affected by “-resource-dir”, we don’t want the tools to see a different standard library (or not find a standard library at all).

Makes sense.

So far in cases where -resource-dir was used for finding libc++, replacing it with “compiler install dir” (“Driver.InstalledDir”) seemed to do the job.

Does “resource-dir used for finding libc++” imply libc++ is in the
resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?

The same recipe seemed to work in that case.
LLVM_ENABLE_PER_TARGET_RUNTIME_DIR relied on -resource-dir before D59013 landed, and seems to use install dir now.

FYI D59013 was reverted because it broke Windows bots, but should re-land soon as D59168.

Hi Petr,

Hi Joel,

Sorry for the late response, was on vacation. Find a few more clarifications inline.

No problem. Thanks.

Hopefully still useful, even though the actual changes have already landed.

There's still more to do. Petr's patch improves the situation for
libcxx, libcxxabi, and libunwind but only when
LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True. Also, we still need to
address openmp issues, which is what my original patch was about.

Hi Ilya,

>
> Hi Joel,
>
> clangd, clang-tidy and other tools do not require to be built from the same revision as the host compiler that the project uses to build code. In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
> However, the internal clang headers (the ones -resource-dir points at) must correspond to the same version of the code that the clang frontend is built from.
> So the aforementioned tools ship their own version of clang's internal headers and pass -resource-dir to the clang frontend to make sure the frontend picks them up. I.e. if the host compiler is also clang, the tools must not pick the host clang's internal headers.
> The tools take other compilation arguments from a compilation database (compile_commands.json).

Thanks. That clears up a lot for me.

Naively, I would've thought any project (clangd, clang-tidy, or any
project external to LLVM) would specify -I to tell the compiler
(clang, gcc, or MSVC) where to find the project's required headers
when building the project. Using -resource-dir sounds like a special
shortcut for the case where the project is clang-based (clangd or
clang-tidy) and the compiler building the project is clang. Is that
right? What do clangd and clang-tidy specify to the compiler if the
compiler is instead gcc or MSVC? Do those have an option equivalent
to -resource-dir?

The underlying "compiler" for clangd and clang-tidy is always clang (as they are calling into the clang frontend), so the tools
would just override -resource-dir as normal. This only works because clang is good enough at understanding gcc flags
and clang-cl is good enough at understanding MVSC flags (although I haven't really checked MSVC flags work, but in
principle they could).

I think I get it. In this use case, is -resource-dir intended as
simply a more convenient -I?

Thanks.

Joel

Hi Petr,

Sorry, I meant Ilya.