I’m maintaining a toolchain targeting a wide range of platforms, including all Apple targets except WatchOS. While upgrading to LLVM 15, I noticed that I’m unable to build compiler-rt for tvOS because building sanitizers and profiling will break it on this platform (due to using fork and bunch of POSIX functionality marked explicitly unavailable on tvOS).
I checked apple/llvm-project but I didn’t find any relevant patches for compiler-rt targeting tvOS.
On Linux-like platforms, I can use -DRUNTIMES_<triple>_BUILD_SANITIZERS to control what gets compiled for a given target. However, on Apple platforms, the triple is always -apple-darwin, making it impossible to distinguish between the targets (that are evidently actually different in capabilities). I don’t want to disable sanitizers on every apple target either, because I’d like to include it whereever I can.
My workaround is configuring the build initially with sanitizers disabled and then just building compiler-rt in a separate directory. Then, after my real distribution build (where tvOS compiler-rt remains disabled and sanitizers etc. enabled), I copy the compiler-rt libs from that directory.
This is obviously not ideal and defeats the purpose of a distribution build.
What could be done to improve this? I think it would be quite natural to call out the concrete OS in the triple, e.g. arm64-apple-darwin-tvos. This probably touches a bunch of moving parts that I’m not aware of and I’m guessing there is good reason for not doing it like that in the first place. So if there is anyone with useful context about this, please let me know.
It is in general, but the Apple runtimes builds are kinda special. All runtimes are built with the same triple, and the build manually adds and removes -m*-version-min and -arch flags to set the OS and arch. It’s also building for multiple OS/arch combinations from a single CMake configure. The -darwin triple is actually important for that because it lets you override the OS and arch via those flags; IIRC the OS-specific triples don’t.
@beanz is probably the best person to help out here.
The -darwin triple is actually important for that because it lets you override the OS and arch via those flags; IIRC the OS-specific triples don’t.
I wonder why this implicit behavior is preferable instead of triples that express the exact target. Setting the deployment target via the environment also overrides the target which makes the env vars difficult to use in a cross-building environment (some builds need the host compiler to compile some tools)
This is an unfortunate but a known problem. The iOS, tvOS, and watchOS sanitizer runtimes cannot build against public SDKs for the reason you gave.
The compiler-rt runtimes are not built the same way as for Linux. For Apple platforms the build is per platform (e.g. macOS, iOS, watchOS, tvOS). For each platform there is a set of associated architectures and they are all built into a “fat binary” containing multiple architectures.
If you are building a Clang toolchain that is trying to target Apple platforms, the compiler-rt runtime should be fat binaries. That is what Clang expects. Doing anything else is asking for trouble.
While this natural for platforms with thin binaries (single architecture per library) it is not natural for platforms with fat binaries (which is what you should be using if you are targeting Apple platforms).
While it might be technically possible re-architecture compiler-rt to build thin binaries for the Apple sanitizer runtime and then afterwards combine them into fat binaries I would heavily recommend not doing this. It would be a huge amount of work for very little gain.
The compiler-rt runtimes are not built the same way as for Linux. For Apple platforms the build is per platform (e.g. macOS, iOS, watchOS, tvOS). For each platform there is a set of associated architectures and they are all built into a “fat binary” containing multiple architectures.
That’s a very fair point and not something I would do differently. I guess my core question here is if there is any way to add some top-level mechanism in CMake that allows per-platform customization of the runtime builds then - so, not full triples like I originally proposed, since you are indeed right that fat binaries are intended to be used on Apple platforms and it would be a can of worms to go against that.
Prefacing with the fact that I don’t work at Apple anymore so my vote doesn’t count here…
I actually had always planned to make the Apple builds generate thin archives for compiler-rt eventually. I didn’t when I setup the CMake build because I was under a lot of pressure to make the CMake build match the autoconf build identically, not just functionally.
When Clang is used as the driver for fat binaries it invokes cc1 and (when linking) ld separately for each slice, then uses lipo to put the thin files into fat files. That means there isn’t really any reason clang needs to have fat archives since the per-architecture linker commands are separate. There is an unknown amount of work to update projects that don’t use build systems that link through Clang, but there aren’t all that many of those historically.
The up-side to using thin binaries is actually pretty big. It would dramatically simplify the compiler-rt build system, but it also simplifies the logic in the clang driver so that the Linux and Darwin driver could actually share more logic.
There is a bunch of work to make this work though, so it is unlikely anyone will ever do it.
What kind of customization do you need? There are already CMake options to enable/disable certain platforms. I don’t think there are options to control which runtimes are enabled (e.g. ASan, UBSan, TSan) on a per platforms basis but it’s probably feasible to add the options if they are needed.
It’s basically what I outlined in my initial post. Since tvOS compiler-rt doesn’t build with sanitizers and profiling enabled, I can either disable those on all of the Apple targets (which I don’t want to) or work around the issue as I am today: building the tvOS compiler-rt in a separate build directory, with sanitizers and profiling disabled; and then copying that compiler-rt into my distribution build install dir. This works, but it’s quite hacky. In terms of (imaginary) CMake options, it would look something like this:
It seems odd that this is only a problem with tvOS. I would have thought the same problem would occur with iOS and watchOS as well.
I think it’s worth noting that tvOS support is off by default.
option(COMPILER_RT_ENABLE_TVOS "Enable building for tvOS - Experimental" Off)
Enabling it means you know what you’re doing so I wouldn’t want it to silently not build things.
That being said options to disable various runtime libraries from being built on a per-platform basis sounds reasonable to me provided it doesn’t overcomplicate the existing code.
It is not something I have time to implement but I can review (or try to find the right people to review) changes you’d like to make.
It’s probably worth noting an alternative to what you propose might be to teach llvm/runtimes to build each Apple platform as a separate external compiler-rt build. It would make builds slower but it would allow per platform control of what is built using existing options in the compiler-rt CMake code.