Should buildbots switch to ENABLE_RUNTIMES instead of ENABLE_PROJECTS for compiler-rt?

Runtimes
1

So I’m looking at unpicking arm/armhf library paths and one part of that business is getting our (Linaro’s) bots back on track.

I see from https://compiler-rt.llvm.org/ that -DLLVM_ENABLE_RUNTIMES= is the recommended way to build compiler-rt.

However, looking at zorg’s zorg/buildbot/builders/ClangBuilder.py the majority of bots with compiler-rt are still adding it to LLVM_ENABLE_PROJECTS.

Is that a problem, is ENABLE_RUNTIMES the future, is there a point where ENABLE_PROJECTS will stop working/bit rot over time?

I wonder which configuration we should be validating.

2

LLVM_ENABLE_PROJECTS=libcxx;libcxxabi;libunwind is deprecated and will soon be unsupported (⚙ D132480 [llvm] Remove libcxx, libcxxabi and libunwind from supported LLVM_ENABLE_PROJECTS).

For compiler-rt, perhaps we can say -DLLVM_ENABLE_RUNTIMES=compiler-rt is strongly recommended, though -DLLVM_ENABLE_PROJECTS=compiler-rt works. GCC occasionally imports compiler-rt into their libsanitizer, I think they only need to build it with trunk GCC.

If -DLLVM_ENABLE_PROJECTS=compiler-rt requires workarounds with some older compiler, I think we can say that is unsupported. Personally I prefer -DLLVM_ENABLE_PROJECTS=compiler-rt stays as that makes my compiler-rt development somewhat easy. I haven’t evaluated how much build system we have to retain the support.

For build bots, it may be a good idea to migrate to -DLLVM_ENABLE_RUNTIMES=compiler-rt, to be consistent with libcxx;libcxxabi;libunwind, and to avoid possible gotcha mixing LLVM_ENABLE_RUNTIMES/LLVM_ENABLE_PROJECTS.

2 Likes
3

I think we should deprecate -DLLVM_ENABLE_PROJECTS=compiler-rt altogether and only support -DLLVM_ENABLE_RUNTIMES=compiler-rt. Note that doing so doesn’t mean that you won’t be able to build compiler-rt with another compiler.

There are two different ways to build runtimes:

  • Runtimes build which is used to build one or more runtimes. To do so, you’d use the following:
    cmake -DLLVM_ENABLE_RUNTIMES=<list of runtimes> <other CMake flags> ${LLVM_SRC_DIR}/runtimes
  • Bootstrapping build which is used to build one or more runtimes with the just built Clang toolchain. It invokes runtimes build as a subbuild. You use it as follows:
    cmake -DLLVM_ENABLE_RUNTIMES=<list of runtimes> <other CMake flags> ${LLVM_SRC_DIR}/llvm

For example, to build compiler-rt with GCC in the runtimes build, you’d use:

cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DLLVM_ENABLE_RUNTIMES=compiler-rt <other CMake flags> ${LLVM_SRC_DIR}/runtimes

Today, you can achieve the same with either:

cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DLLVM_ENABLE_PROJECTS=compiler-rt <other CMake flags> ${LLVM_SRC_DIR}/llvm

or:

cmake -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ <other CMake flags> ${LLVM_SRC_DIR}/compiler-rt

The main advantage of using the runtimes build is that we can avoid the CMake duplication between different runtimes builds and simplify maintenance.

For example, in libcxx, libcxxabi and libunwind, we had similar logic duplicated three times. That’s because someone would write it first in one of the runtimes, and then someone else would copy & paste it into other two, but then others would start making changes and wouldn’t often replicate those changes in every copy, and we would end up with similar but different copies making maintenance a nightmare. With the runtimes build, we can deduplicate all of that logic and have only a single copy that’s shared by all runtimes.

You can already build compiler-rt with -DLLVM_ENABLE_RUNTIMES=compiler-rt, but we can’t start removing a lot of this complexity until we deprecate -DLLVM_ENABLE_PROJECTS=compiler-rt like we did for libcxx, libcxxabi and libunwind. I believe this would be a huge improvement.

For example, compiler-rt build already supports building tests with the just built Clang, but rather than relying on the bootstrapping build, it does so by manually invoking Clang as a custom command which means that a lot of the standard flag handling provided by CMake doesn’t work, and it requires developers to use various workarounds.

Note that we may not be quite ready to deprecate -DLLVM_ENABLE_PROJECTS=compiler-rt yet. Even though -DLLVM_ENABLE_RUNTIMES=compiler-rt has been supported for some time, it’s not widely used—most notably, it’s not used by buildbots—so there might be scenarios that are broken, but it would be great if we can start promoting its use so we find and address all the issues, and aim for deprecation and removal of -DLLVM_ENABLE_PROJECTS=compiler-rt ideally after the LLVM 16 branch point.

Updating all buildbots to use -DLLVM_ENABLE_RUNTIMES=compiler-rt would be a first great step towards the eventual deprecation.

1 Like
4

I was going to say that test-release.sh probably also needs updating, but it seems that already happened :slight_smile: ⚙ D112748 [release] Use the Bootstrapping build for building LLVM releases

I worry that we’re moving to a “the current way is deprecated and the new way isn’t working yet” situation. For example,

$ cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS='clang;lld' -DLLVM_ENABLE_RUNTIMES=compiler-rt ../llvm
$ ninja check-asan   
ninja: error: unknown target 'check-asan', did you mean 'check-clang'?


$ cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS='clang;lld;compiler-rt' ../llvm
$ ninja check-asan
(runs the tests)
5

Yeah we shipped 14.x with this and now also 15.x has been using this since the start. But I don’t think the Windows batch script has been updated to do this. We should make sure that works before 16.x.

6

Sounds good to me.

I will start this process and will specifically handle Linaro’s bots.

If it’s as simple as moving some cmake options around, I should be able to add an option to zorg to put certain projects in runtimes instead of projects. That can be opt in initially, changing to an opt out as other bot maintainers validate it works for them.

Once I have a change for review I’ll also post here on discourse and include the maintainers of other bots that use compiler-rt.

This is why I asked this question in fact, which gives me hope that the Arm/AArch64 side should be ok if we’ve managed to build releases for them.

7

There is one lingering issue that I know about with the runtime builds - check-all didn’t run all the runtime tests - this was fixed by @petrhosek - but then reverted because of a issue:

:gear: D132438 [runtimes] Use a response file for runtimes test suites (llvm.org)

This should be fixed on main as soon as it’s possible especially if the bots are moving to use this method. If this patch is not landed all the tests are not executed correctly, which is not great.

8

I’ve made a note to give this a try once 15 has shipped.

1 Like
9

Sounds good. I think deprecating -DLLVM_ENABLE_PROJECTS=compiler-rt is fine (we need help from build bot maintainers to migrate their use cases and ensure LLVM_ENABLE_RUNTIMES=compiler-rt works for them).

10

I gave this a go and quickly ran into problems (`check-compiler-rt` fails with various errors when using the runtimes build on windows · Issue #58568 · llvm/llvm-project · GitHub for a start) which seems consistent with the “the working way is deprecated, and the new way isn’t ready yet” situation (s/Google/llvm/ here).

Happy to give it another try in the future if the support improves.

11

Thanks for the info, I have also been watching ⚙ D132438 [runtimes] Use a response file for runtimes test suites. Until the issues are ironed out there our (Linaro’s) bots will continue using PROJECTS.

12

All Linaro’s bots are now building compiler-rt as a runtime and have had no issues with it so far.

I will be contacting other bot owners over the coming days to see if they would like to do the same. If not or I am unable to find a contact, I will update their config to explicitly disable the use of runtimes. Then finally the default(s) can be flipped.

1 Like