Use of host/target compiler when building compiler-rt

I stumbled across what seems to be a bug (to me) in the compiler-rt build:

The compiler-rt libraries themselves are built with the host compiler while the tests are built and then linked with the just-built clang.

It was my understanding that the goal/intent/need was to have the compiler-rt library build with the just-built clang? Did I misunderstand that?*

Sterling: Chandler seemed to think you might be interested in this issue & possibly addressing it given you’re working on compiler-rt bring-up? It’d probably be useful to have compiler-rt built with the just-built clang for performance reasons.

Evgeniy - not sure if you’re interested in this or have much context? Know if this is right/wrong/neutral, etc?

  • reasons include performance, ABI compatibility, etc (I thought this was necessary for correctness in some way) - also, otherwise it seems excessive to hold up the whole build on waiting for just-built clang to finish, then use that to compile some tests. (well, I realize some of the tests are end-to-end, so they do need the just-built compiler)

Yes, this is a aspect of the larger problem that clang bootstrap doesn’t work for a cross-compiler. The build (mostly?) assumes that host==target during the build of clang itself, and then if you want another architecture also, you run a second build of the target libraries, and manually merge the trees.

If you think about compiler-rt as being compiled for the target rather than the host, the problem you describe here is exactly the same one, and we have been getting lucky.

At the moment, the blaze builds of clang do exactly the procedure described above, so this hasn’t been a terrible problem for Google, but I do think it is something that should be fixed (I’m working on another aspect of compiler-rt bringup at the moment, so won’t solve this in the immediate future.)

gnu systems have a make variable, “CC_FOR_TARGET” that addresses this problem. I imagine llvm should adopt a similar mechanism inside cmake.

Yes, this is a aspect of the larger problem that clang bootstrap doesn’t work for a cross-compiler. The build (mostly?) assumes that host==target during the build of clang itself, and then if you want another architecture also, you run a second build of the target libraries, and manually merge the trees.

I kind of roughly follow that, but not too well.

If you think about compiler-rt as being compiled for the target rather than the host, the problem you describe here is exactly the same one, and we have been getting lucky.

Sure - if a PPC clang is being built from an x86 host, how would compiler-rt be built (OK, it could be built with the just-built clang, which it isn’t at the moment) and tested (can’t really be tested because the host can’t run PPC binaries).

At the moment, the blaze builds of clang do exactly the procedure described above, so this hasn’t been a terrible problem for Google, but I do think it is something that should be fixed (I’m working on another aspect of compiler-rt bringup at the moment, so won’t solve this in the immediate future.)

Rightio

gnu systems have a make variable, “CC_FOR_TARGET” that addresses this problem. I imagine llvm should adopt a similar mechanism inside cmake.

Not sure I follow on the need/use of CC_FOR_TARGET compared to using the just-built clang as the CC_FOR_TARGET (which it seems we have some plumbing for already - the just-built clang is used for building the compiler-rt tests, but not for building the library. I /think/ it should be used for both)

  • Dave

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

  2. The new way, is to place compiler-rt under LLVM/runtimes. In this path the build system will automatically build with the just-built compiler. This path also splits compiler-rt into two separate build steps, one that configures and builds the builtins with the just-built compiler, and a second that configures and builds the sanitizer libraries.

The second path also works for many (but not all) of our other runtime library projects. I know it works for libcxx, libcxxabi, and libunwind. Petr Hosek (CC’d) has also been working on support for multi-arch builtin and runtime library builds so that you can generate full cross-compilers from a single cmake invocation.

-Chris

I think the long term plan is to build compiler-rt with just build
clang. Nothing else makes sense, really. AFAIK that's what
LLVM_BUILD_EXTERNAL_COMPILER_RT is for.

Why is this not the default? Thanks again, Hal

Two reasons. (1) It is buggy, and nobody fixed the issues because (2) the long-term plan is to not support building compiler-rt (or any other runtime library) under llvm/projects. The goal is to migrate entirely to llvm/runtimes where the support is more complete.

-Chris

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

I thought the BUILD_EXTERNAL variables were for use when projects were not embedded within the llvm source tree (mostly in use by Takumi’s flat buildbots that checout the top-level project without embedding, say, clang or compiler-rt within the llvm source tree)?

  1. The new way, is to place compiler-rt under LLVM/runtimes. In this path the build system will automatically build with the just-built compiler. This path also splits compiler-rt into two separate build steps, one that configures and builds the builtins with the just-built compiler, and a second that configures and builds the sanitizer libraries.

Huh, OK - could someone remove the legacy format, then? If it’s a trap.

That said, I tried putting compiler-rt in runtimes instead of projects and I got a bunch of cmake errors starting with:

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “GotsanRuntimeCheck” of target “check-runtimes” does
not exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

Any ideas?

For additional context, please see this thread from August:
http://lists.llvm.org/pipermail/llvm-dev/2016-August/104174.html

-Chris

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

I thought the BUILD_EXTERNAL variables were for use when projects were not embedded within the llvm source tree (mostly in use by Takumi’s flat buildbots that checout the top-level project without embedding, say, clang or compiler-rt within the llvm source tree)?

You are confusing this with the similarly named LLVM_EXTERNAL_${nameUPPER}_SOURCE_DIR variables.

  1. The new way, is to place compiler-rt under LLVM/runtimes. In this path the build system will automatically build with the just-built compiler. This path also splits compiler-rt into two separate build steps, one that configures and builds the builtins with the just-built compiler, and a second that configures and builds the sanitizer libraries.

Huh, OK - could someone remove the legacy format, then? If it’s a trap.

I’m not sure the new path is fully supported in every workflow, so removing it seems like a not great idea at the moment.

That said, I tried putting compiler-rt in runtimes instead of projects and I got a bunch of cmake errors starting with:

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “GotsanRuntimeCheck” of target “check-runtimes” does
not exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

Any ideas?

I have never encountered that issue. It looks like the tsan test code is out of sync. If you go into tsan/test/CMakeLists.txt and on Line 2 add this to the if statement “AND TARGET GotsanRuntimeCheck” that should fix the issue.

-Chris

From a compiler developers point of view I wanted to stress that we at least should keep the option not to do it, as in my experience development workflow heavily slows down if you have to wait around for compiler-rt to rebuild after every small change you make to the compiler.

Of course I agree that we should switch the default to something that uses the just-built compiler so people unaware of these issues get the safe and correct thing.

  • Matthias

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

I thought the BUILD_EXTERNAL variables were for use when projects were not embedded within the llvm source tree (mostly in use by Takumi’s flat buildbots that checout the top-level project without embedding, say, clang or compiler-rt within the llvm source tree)?

You are confusing this with the similarly named LLVM_EXTERNAL_${nameUPPER}_SOURCE_DIR variables.

Ah, right - indeed.

  1. The new way, is to place compiler-rt under LLVM/runtimes. In this path the build system will automatically build with the just-built compiler. This path also splits compiler-rt into two separate build steps, one that configures and builds the builtins with the just-built compiler, and a second that configures and builds the sanitizer libraries.

Huh, OK - could someone remove the legacy format, then? If it’s a trap.

I’m not sure the new path is fully supported in every workflow, so removing it seems like a not great idea at the moment.

That said, I tried putting compiler-rt in runtimes instead of projects and I got a bunch of cmake errors starting with:

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “GotsanRuntimeCheck” of target “check-runtimes” does
not exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

Any ideas?

I have never encountered that issue. It looks like the tsan test code is out of sync. If you go into tsan/test/CMakeLists.txt and on Line 2 add this to the if statement “AND TARGET GotsanRuntimeCheck” that should fix the issue.

Hrm - not sure which CMakeLists.txt you’re referring to? In my runtimes/compiler-rt/lib/tsan/tests/CMakeLists.txt the first few lines are:

include_directories(…/rtl)

add_custom_target(TsanUnitTests)
set_target_properties(TsanUnitTests PROPERTIES
FOLDER “TSan unittests”)

no if condition I could modify?

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

Why is this not the default?

Two reasons. (1) It is buggy, and nobody fixed the issues because (2) the long-term plan is to not support building compiler-rt (or any other runtime library) under llvm/projects. The goal is to migrate entirely to llvm/runtimes where the support is more complete.

From a compiler developers point of view I wanted to stress that we at least should keep the option not to do it, as in my experience development workflow heavily slows down if you have to wait around for compiler-rt to rebuild after every small change you make to the compiler.

If that’s a valid option, I’d be open to it - it does really slow down the dev cycle & it’s one of the reasons I often end up moving compiler-rt out/not building it anymore.

I’m not sure if that’s an option at the moment - if it is, I’d love to know how to do that & it might at least unblock my use case anyway, even if there’s still a need for a broader fix.

The legacy way (I wasn’t aware it is legacy) of “-DLLVM_BUILD_EXTERNAL_COMPILER_RT=On/Off” has worked fine for me so far. I haven’t tried the runtimes directory style yet, but it looks like a nicer more explicit way than specifying cmake flags and is just suffering from a bit probably?

  • Matthias

It would be really good to update https://compiler-rt.llvm.org/ with more detail on how to build this in a supported way.

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

Why is this not the default?

Two reasons. (1) It is buggy, and nobody fixed the issues because (2) the long-term plan is to not support building compiler-rt (or any other runtime library) under llvm/projects. The goal is to migrate entirely to llvm/runtimes where the support is more complete.

From a compiler developers point of view I wanted to stress that we at least should keep the option not to do it, as in my experience development workflow heavily slows down if you have to wait around for compiler-rt to rebuild after every small change you make to the compiler.

If that’s a valid option, I’d be open to it - it does really slow down the dev cycle & it’s one of the reasons I often end up moving compiler-rt out/not building it anymore.

I’m not sure if that’s an option at the moment - if it is, I’d love to know how to do that & it might at least unblock my use case anyway, even if there’s still a need for a broader fix.

The legacy way (I wasn’t aware it is legacy) of “-DLLVM_BUILD_EXTERNAL_COMPILER_RT=On/Off” has worked fine for me so far.

Hmm, it’s set to “OFF” for me, but the unit tests are still compiled/linked with the just-built clang. Is that not the case for you?

I’m assuming it was setup that way because the compiler-rt for the sanitizers are end-to-end and so need the just-built clang to be able to build a sanitized binary, etc. So I’m not sure how this could work /entirely/ using the host compiler and never using the just-built compiler.

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

I thought the BUILD_EXTERNAL variables were for use when projects were not embedded within the llvm source tree (mostly in use by Takumi’s flat buildbots that checout the top-level project without embedding, say, clang or compiler-rt within the llvm source tree)?

You are confusing this with the similarly named LLVM_EXTERNAL_${nameUPPER}_SOURCE_DIR variables.

Ah, right - indeed.

  1. The new way, is to place compiler-rt under LLVM/runtimes. In this path the build system will automatically build with the just-built compiler. This path also splits compiler-rt into two separate build steps, one that configures and builds the builtins with the just-built compiler, and a second that configures and builds the sanitizer libraries.

Huh, OK - could someone remove the legacy format, then? If it’s a trap.

I’m not sure the new path is fully supported in every workflow, so removing it seems like a not great idea at the moment.

That said, I tried putting compiler-rt in runtimes instead of projects and I got a bunch of cmake errors starting with:

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “GotsanRuntimeCheck” of target “check-runtimes” does
not exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

Any ideas?

I have never encountered that issue. It looks like the tsan test code is out of sync. If you go into tsan/test/CMakeLists.txt and on Line 2 add this to the if statement “AND TARGET GotsanRuntimeCheck” that should fix the issue.

Hrm - not sure which CMakeLists.txt you’re referring to? In my runtimes/compiler-rt/lib/tsan/tests/CMakeLists.txt the first few lines are:

Sorry, I meant /test/tsan/CMakeLists.txt, not tsan/test.

-Chris

David,

This is an area that has had a lot of development over the last two years.

There are two supported ways in the LLVM build system to build compiler-rt with the just-built compiler.

  1. The legacy way is for if compiler-rt is under LLVM/projects. You can specify -DLLVM_BUILD_EXTERNAL_COMPILER_RT=On, which will configure compiler-rt using the just-built clang after clang is built.

I thought the BUILD_EXTERNAL variables were for use when projects were not embedded within the llvm source tree (mostly in use by Takumi’s flat buildbots that checout the top-level project without embedding, say, clang or compiler-rt within the llvm source tree)?

You are confusing this with the similarly named LLVM_EXTERNAL_${nameUPPER}_SOURCE_DIR variables.

Ah, right - indeed.

  1. The new way, is to place compiler-rt under LLVM/runtimes. In this path the build system will automatically build with the just-built compiler. This path also splits compiler-rt into two separate build steps, one that configures and builds the builtins with the just-built compiler, and a second that configures and builds the sanitizer libraries.

Huh, OK - could someone remove the legacy format, then? If it’s a trap.

I’m not sure the new path is fully supported in every workflow, so removing it seems like a not great idea at the moment.

That said, I tried putting compiler-rt in runtimes instead of projects and I got a bunch of cmake errors starting with:

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “GotsanRuntimeCheck” of target “check-runtimes” does
not exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

Any ideas?

I have never encountered that issue. It looks like the tsan test code is out of sync. If you go into tsan/test/CMakeLists.txt and on Line 2 add this to the if statement “AND TARGET GotsanRuntimeCheck” that should fix the issue.

Hrm - not sure which CMakeLists.txt you’re referring to? In my runtimes/compiler-rt/lib/tsan/tests/CMakeLists.txt the first few lines are:

Sorry, I meant /test/tsan/CMakeLists.txt, not tsan/test.

Think that got me past that error, but (sorry I wasn’t clear) it was only one of many errors. Here are the next… 5, say…

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “TsanUnitTests” of target “check-runtimes” does not
exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “cfi” of target “check-runtimes” does not exist.
Call Stack (most recent call first):
CMakeLists.txt:110 (add_lit_target)

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “TsanUnitTests” of target “check-compiler-rt” does
not exist.
Call Stack (most recent call first):
compiler-rt/test/CMakeLists.txt:94 (add_lit_target)

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “cfi” of target “check-compiler-rt” does not exist.
Call Stack (most recent call first):
compiler-rt/test/CMakeLists.txt:94 (add_lit_target)

CMake Error at /usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1174 (add_dependencies):
The dependency target “TsanUnitTests” of target “check-tsan” does not
exist.
Call Stack (most recent call first):
/usr/local/google/home/blaikie/dev/llvm/build/clang/debug/split/notypes/nostandalone/lib/cmake/llvm/AddLLVM.cmake:1195 (add_lit_target)
compiler-rt/test/tsan/CMakeLists.txt:46 (add_lit_testsuite)

I’m assuming there’s some systemic problem with the way I’m holding this (if it’s working for other people)?

  • Dave

I’ll try and reproduce later today. Is this Linux? Can you give me your CMake command line?

-Chris

I’ll try and reproduce later today. Is this Linux? Can you give me your CMake command line?

Excuse the delay, been busy setting up a new machine - also an opportunity to try clean cmake setups rather than my aging configurations that have a bunch of old stuff baked in and manual variables changed, etc. (& I do so wish that CMakeCache.txt had a comment at teh top describing the command that produced it like the config.log - that way I’d probably be more likely to copy/paste and modify the command line than hand editing the cache file)

Anyway, so I don’t have the compounding issue (local libstdc++ install with ABI incompatibility to my system libstdc++) so I can build/test compiler-rt when it’s in projects at the moment (though I will soon locally install GCC 6.3 and have the problems with xray again).

But when I place compiler-rt in runtimes I do have problems.

Here’s my cmake command:

cmake -G Ninja -DCMAKE_INSTALL_PREFIX=$HOME/dev/llvm/install -DCMAKE_BUILD_TYPE=Debug …/…/src/ -DCMAKE_C_COMPILER=$HOME/install/bin/clang -DCMAKE_CXX_COMPILER=$HOME/install/bin/clang++ -DLLVM_USE_SPLIT_DWARF=ON -DLLVM_ENABLE_WERROR=ON -DLLVM_BUILD_EXAMPLES=ON -DLLVM_INCLUDE_GO_TESTS=OFF

& here’s the result of “ninja check-all”:

[3671/3683] cd /usr/local/google/home/blaikie/dev/llvm/build/default…fault/runtimes/runtimes-bins/ --target check-runtimes --config Debug
[318/409] Generating tsan_mman_test.cc.x86_64.o
FAILED: compiler-rt/lib/tsan/tests/unit/tsan_mman_test.cc.x86_64.o
cd /usr/local/google/home/blaikie/dev/llvm/build/default/runtimes/runtimes-bins/compiler-rt/lib/tsan/tests/unit && /usr/local/google/home/blaikie/dev/llvm/build/default/./bin/clang -fPIC -fvisibility-inlines-hidden -Werror -Werror=date-time -std=c++11 -fcolor-diagnostics -Wall -Werror -std=c++11 -Wno-unused-parameter -Wno-unknown-warning-option -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fvisibility-inlines-hidden -fno-lto -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -Wno-non-virtual-dtor -fPIE -fno-rtti -Wno-covered-switch-default -DGTEST_NO_LLVM_RAW_OSTREAM=1 -DGTEST_HAS_RTTI=0 -I/usr/local/google/home/blaikie/dev/llvm/src/utils/unittest/googletest/include -I/usr/local/google/home/blaikie/dev/llvm/src/utils/unittest/googletest -I/usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib -I/usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib/tsan/rtl -DGTEST_HAS_RTTI=0 -m64 -c -o tsan_mman_test.cc.x86_64.o /usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib/tsan/tests/unit/tsan_mman_test.cc
/usr/local/google/home/blaikie/dev/llvm/src/runtimes/compiler-rt/lib/tsan/tests/unit/tsan_mman_test.cc:14:10: fatal error: ‘sanitizer/allocator_interface.h’ file not found
#include <sanitizer/allocator_interface.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
[370/409] Generating ASAN_INST_TEST_OBJECTS.asan_test.cc.x86_64-inline.o