Compiler support in libc++

Project Infrastructure LLVM Dev List Archives
21

Fantastic! It looks like the cmake option I was suggesting already exists actually :slight_smile:

I had to read our CMake scripts to figure out how to use it though, looking at https://llvm.org/docs/BuildingADistribution.html#relevant-cmake-options it mentions the CMake option, but just running locally ninja does not get them to be built, I had to explicitly call ninja runtimes to get libc++ to be built, I don’t know if this is intended?
I also need to explicitly have -DLLVM_ENABLE_PROJECTS=clang by the way otherwise ninja runtimes will error out obscurely at some point, the cmake handling of this option isn’t defensive about it at the moment.

Anyway, I need to update the bot config to see if this “just works”, but locally it seems promising!

Thanks Michael!

22

Hi,

It seems to me that this would require one extra stage of bootstrap in CI for many buildbots.
For example, today I have a Linux bot with a clang-8 host compiler and libstdc++. The goal is to ensure that MLIR (but it is applicable to any project) builds with clang and libc++ at the top of the main branch.
So the setup is:

  • stage1: build clang/libc++ with host clang-8/libstdc++
  • stage2: build test “anything” using stage1 (ninja check-all in the monorepo for example, but applicable to any other external project)

With this proposal, the setup would be:

  • stage1: build just clang with host clang-8/libstdc++

  • stage2: build clang/libc++ with stage1 clang and host libstdc++

  • stage3: build test “anything” using stage2 (ninja check-all in the monorepo for example, but applicable to any other external project)

Would it be possible to change the build system so that libc++ can be built like compiler-rt, using the just-built clang? That would then avoid the need for the extra stage? (though it would bottleneck the usual build a bit - not being able to start the libc++ build until after clang build)

That’s a good point:

  • stage1: build just clang with host clang-8/libstdc++
  • stage1.5: build libc++ with stage1 clang
  • stage 2: assemble toolchain with clang from stage1 and libc++ from stage2
  • stage3: build test “anything” using stage2 (ninja check-all in the monorepo for example, but applicable to any other external project)

Since this “stage 2” is the new “stage1”, I believe that this should be made completely straightforward to achieve. Ideally it should boil down to a single standard CMake invocation to produce this configuration.

I think the Runtimes build is exactly what you’re looking for. With the runtimes build, you say:

$ cmake -S “${MONOREPO_ROOT}/llvm” -B "${BUILD_DIR}”
-DLLVM_ENABLE_PROJECTS="clang”
-DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi”
-DLLVM_RUNTIME_TARGETS="x86_64-unknown-linux-gnu”

And then you can just do:

$ make -C $BUILD_DIR cxx

That will bootstrap Clang and then build libc++ with the just-built Clang. I don’t know whether you consider that to be one or two stages, but it happens automatically in that single CMake invocation. And since building libc++ is basically trivial, this takes approximately the same time as building Clang only.

Yes, thanks, this config is just perfectly fitting here!

& again, this isn’t so much a proposal of change, but one of documenting the current state of things - which reveals the current situations are sort of unsupported? (though it also goes against the claim that they’re untested) - so I’ll be curious to hear from the libc++ folks about this for sure.

Right: I’m absolutely not convinced by the “we’re documenting the current state of things” actually.
In particular my take in general on what we call “supported” is a policy that “we revert if we break a supported configuration” and “we accept patches to fix a supported configuration”. So the change here is that libc++ would not accept to revert when they break an older toolchain, and we wouldn’t accept patches to libc++ to fix it.
We don’t necessarily have buildbots for every configuration that we claim LLVM is supporting, yet this is the policy, and I’m quite wary of defining the “current state of things” based exclusively on the current public buildbots setup.

To be clear, what we do today to “fix” older compilers is usually to mark failing tests in the test suite with XFAIL or UNSUPPORTED annotations. We don’t actually provide a good level of support for those compilers. There’s also other things that we simply can’t fix, like the fact that a libc++ built with a compiler that doesn’t know about char8_t (for example) won’t produce the RTTI for char8_t in the dylib, and hence will produce a dylib where some random uses of char8_t will break down. This is just an example, but my point is that it’s far better to clarify the support policy to something that we know will work, and that we can commit to supporting. There’s a small upfront cost for people running build bots right now, but once things are setup it’ll just be better for everyone.

The only way to avoid adding a stage in the bootstrap is to keep updating the bots with a very recent host clang (I’m not convinced that increasing the cost of maintenance for CI / infra is good in general).

We should aim for a better balance: it is possible that clang-5 is too old (I don’t know?), but there are people (like me, and possibly others) who are testing HEAD with older compiler (clang-8 here) and it does not seem broken at the moment (or the recent years), I feel there should be a strong motivation to break it.

Libc++ on Clang 8 doesn’t look broken because it builds. And it builds because you’ve been pinging us on Phabricator when we break you with a change, and we add a “workaround” that makes it build. But there’s no guarantee about the “quality" of the libc++ that you get in that case though. That’s exactly what we want to avoid - you get something that “kinda works”, yet we still have to insert random workarounds in the code. It’s a lose/lose situation.

To be fair there has been exactly one breakage caused by libc++ over the last 2 years: while there may be issue in corner cases like you mentions, it seems to work fine for many projects (including clang/llvm/mlir bootstrap since this is what I’ve been testing).

23

<llvm-dev@lists.llvm.org>:

That’s a good point:

  • stage1: build just clang with host clang-8/libstdc++
  • stage1.5: build libc++ with stage1 clang
  • stage 2: assemble toolchain with clang from stage1 and libc++ from stage2
  • stage3: build test “anything” using stage2 (ninja check-all in the monorepo for example, but applicable to any other external project)

Stage 1.5 is exactly what cmake
-DLLVM_ENABLE_RUNTIMES=libcxxabi;libcxx should do.

Fantastic! It looks like the cmake option I was suggesting already exists actually :slight_smile:

I had to read our CMake scripts to figure out how to use it though,

Documentation is definitely something that needs improving since right now it’s basically completely absent.

looking at https://llvm.org/docs/BuildingADistribution.html#relevant-cmake-options it mentions the CMake option, but just running locally ninja does not get them to be built, I had to explicitly call ninja runtimes to get libc++ to be built, I don’t know if this is intended?

In all our examples, we always use ninja distribution and include runtimes in LLVM_DISTRIBUTION_COMPONENTS and I haven’t thought of also including runtimes in the default target, but it makes sense and should be easy to fix.

I also need to explicitly have -DLLVM_ENABLE_PROJECTS=clang by the way otherwise ninja runtimes will error out obscurely at some point, the cmake handling of this option isn’t defensive about it at the moment.

That’s also something we could improve.

Anyway, I need to update the bot config to see if this “just works”, but locally it seems promising!

The runtimes build is under active development and we’re interested in hearing about issues so please let me know if you run into problems.

24

<llvm-dev@lists.llvm.org>:

That’s a good point:

  • stage1: build just clang with host clang-8/libstdc++
  • stage1.5: build libc++ with stage1 clang
  • stage 2: assemble toolchain with clang from stage1 and libc++ from stage2
  • stage3: build test “anything” using stage2 (ninja check-all in the monorepo for example, but applicable to any other external project)

Stage 1.5 is exactly what cmake
-DLLVM_ENABLE_RUNTIMES=libcxxabi;libcxx should do.

Fantastic! It looks like the cmake option I was suggesting already exists actually :slight_smile:

I had to read our CMake scripts to figure out how to use it though,

Documentation is definitely something that needs improving since right now it’s basically completely absent.

looking at https://llvm.org/docs/BuildingADistribution.html#relevant-cmake-options it mentions the CMake option, but just running locally ninja does not get them to be built, I had to explicitly call ninja runtimes to get libc++ to be built, I don’t know if this is intended?

In all our examples, we always use ninja distribution and include runtimes in LLVM_DISTRIBUTION_COMPONENTS and I haven’t thought of also including runtimes in the default target, but it makes sense and should be easy to fix.

I also need to explicitly have -DLLVM_ENABLE_PROJECTS=clang by the way otherwise ninja runtimes will error out obscurely at some point, the cmake handling of this option isn’t defensive about it at the moment.

That’s also something we could improve.

Anyway, I need to update the bot config to see if this “just works”, but locally it seems promising!

The runtimes build is under active development and we’re interested in hearing about issues so please let me know if you run into problems.

I will be changing the default way of building libc++ to the runtimes build (in the documentation). I am also in the process of adding a CI job to build and test libc++ with a bootstrapped Clang. Once those configurations are the default ones, we should be able to fix issues more easily as the usage of the runtimes build will increase. I do agree that it’s a bit difficult to find documentation on it right now though :-).

Louis

25

I think it’s reasonable to raise the compiler version floor for libc++, but I think I would like to see a more relaxed policy with respect to clang. Maybe the last two releases of clang, so that a user of ToT libc++ with stable clang doesn’t have to rush to upgrade clang as soon as it is released. If you support the last two releases, the user always has six months of lead time before updating, and libc++ never supports a compiler older than a year.

I’ll also point out that, I see a lot of support on this thread, but I see a lot of developer representation, and not much user representation. I have no idea how to effectively survey users of libc++, though.

Lastly, from Chromium’s PoV, Chromium has an ancient NaCl toolchain, and we believe we may be using ToT libc++ with it. We have other reasons (C++17 for one) to want to either remove or update this compiler, so please don’t consider this a blocker for libc++. I only mention it to show that users do sometimes inadvertently develop dependencies on old compilers.

26

I think it's reasonable to raise the compiler version floor for libc++, but I think I would like to see a more relaxed policy with respect to clang. Maybe the last two releases of clang, so that a user of ToT libc++ with stable clang doesn't have to rush to upgrade clang as soon as it is released. If you support the last two releases, the user always has six months of lead time before updating, and libc++ never supports a compiler older than a year.

I'll also point out that, I see a lot of support on this thread, but I see a lot of developer representation, and not much user representation. I have no idea how to effectively survey users of libc++, though.

+1.

From user POV, supporting only the last two stable clang releases

is *the smallest reasonable guarantee*.

Roman

27

On MacPorts, I maintain the llvm/clang/flang/libc++ ports for darwin systems.

We support thousands of users, all of whom open tickets for any issues, which we resolve as we go along.

The current tip of trunk runs on all darwin systems back to and including 10.6.8.

We bootstrap from system roots to trunk on all them, using stepping stone clang versions along the way.

If there is anything you would like to know about this process or questions raised thereof, please ask.

Best,

Ken

28

“Users” are just going to use a toolchain distribution put together by someone else, right?

I’d expect the person putting together such a toolchain distribution to download a release of llvm, and build all of the components from that same revision. It’s historically been annoying to ensure that you actually build the runtime libraries using the just-built clang, when you’re building a set of llvm+clang+compiler-rt+libcxxabi+libcxx all together, instead of whatever compiler you had lying around…but once the documentation and process is updated to make the right thing happen in the “obvious” path, ISTM that solves 99% of the problem here.

For developers of libc++ who are making changes against devhead, they may want to avoid rebuilding clang every time they want to test a new revision of libc++. For that, the “last stable” promise seems useful. But that seems like it shouldn’t really affect users?

The question is whether there’s circumstances in which someone who is putting together a toolchain distribution needs to upgrade to a newer version of libc++, yet remain on an older release of clang (…but only up to 1 year old). If that’s what folks are saying is necessary: maybe someone can help explain why? It doesn’t seem like it should be needed, to me.

29

Sorry for the late reply, I’m on vacation right now and until the end of March. See my answers below.

“Users” are just going to use a toolchain distribution put together by someone else, right?

Yes, that would be my expectation too. I guess that depends how you define users.

I’d expect the person putting together such a toolchain distribution to download a release of llvm, and build all of the components from that same revision. It’s historically been annoying to ensure that you actually build the runtime libraries using the just-built clang, when you’re building a set of llvm+clang+compiler-rt+libcxxabi+libcxx all together, instead of whatever compiler you had lying around…but once the documentation and process is updated to make the right thing happen in the “obvious” path, ISTM that solves 99% of the problem here.

I agree, I think this solves 99% of problems.

For developers of libc++ who are making changes against devhead, they may want to avoid rebuilding clang every time they want to test a new revision of libc++. For that, the “last stable” promise seems useful. But that seems like it shouldn’t really affect users?

The question is whether there’s circumstances in which someone who is putting together a toolchain distribution needs to upgrade to a newer version of libc++, yet remain on an older release of clang (…but only up to 1 year old). If that’s what folks are saying is necessary: maybe someone can help explain why? It doesn’t seem like it should be needed, to me.

This is useful if you ship libc++ as part of a different product than the product you ship Clang in, yet both need to interoperate. For example, we do this on Apple platforms by means of shipping Clang in Xcode, but shipping libc++ as part of the operating system and corresponding SDK. In this case, the “last stable release” guarantee means that everything will work as long as Clang keeps getting updated at a reasonable rate.

I think it’s reasonable to raise the compiler version floor for libc++, but I think I would like to see a more relaxed policy with respect to clang. Maybe the last two releases of clang, so that a user of ToT libc++ with stable clang doesn’t have to rush to upgrade clang as soon as it is released. If you support the last two releases, the user always has six months of lead time before updating, and libc++ never supports a compiler older than a year.

Hmm, yes, I think that would make sense. This would indeed give more leeway to folks testing ToT libc++ with a released Clang, so that they wouldn’t need to update the Clang on their CI fleet the very second a new Clang gets released (if they don’t, the next libc++ commit could technically break them). I think that makes sense - at least we can start with that and see if we want to make it more stringent in the future.

I’ll be creating a Phabricator review to enshrine this updated policy when I come back from vacation, and it will start being enforced at the next release.

Louis

30

Is the outcome of this discussion documented anywhere? I only found Getting Started with the LLVM System — LLVM 16.0.0git documentation with generic toolchain requirements for LLVM, but not any more specific information on libcxx.

The reason I’m looking for this is that ⚙ D119697 [libunwind] Only include cet.h if __CET__ defined claims the increased toolchain requirements also apply to libunwind, which doesn’t seem to match what was proposed here.

31

You can find the platforms and compilers supported by libc++ and libc++abi here: “libc++” C++ Standard Library — libc++ 16.0.0git documentation.

Until now, I would have thought/assumed that it applies to libunwind as well, because we always treat the three projects similarly. I think all the runtimes should be using the same requirements. However, if folks want to support older compilers for libunwind alone, we can have that discussion. I’ll be strongly against because it would complicate all of our testing infrastructure, which we’re only just managing to keep under control. But that’s just my opinion, we can still have that discussion.

32

Thanks! I was looking on Building libc++ — libc++ documentation, while the information was right there on the frontpage…

We already have differing build requirements for runtimes, in particular compiler-rt (as far as I know) uses the same toolchain baseline as the rest of LLVM, not the libcxx/libcxxabi baseline. I would expect that libunwind would get the same treatment as compiler-rt, not as libcxx.

There is a very strong case for requiring a recent toolchain for libcxx, because a) it’s a C++ standard library, so naturally needs a compiler supporting latest C++ and b) if you’re building libcxx, you probably also want to build clang (or have built it before).

I don’t think that either of those are true for libunwind. As far as I know, there is no technical reason why it needs a recent C++ version (beyond the general arguments that could be applied to any code in LLVM, and as such aren’t relevant here.)

Additionally, libunwind is like compiler-rt in that it can be used as a runtime library for non-C++ compilers. For example, rust uses llvm-libunwind on some platforms, but obviously has no interest in building clang (or a very recent gcc).

33

I’d actually look at it the other way around. Why do you want/need to support an older compiler for libunwind? You’re building a toolchain – you might as well use the bootstrapping build and use a tip-of-trunk compiler.

Also, the reason why compiler-rt isn’t treated uniformly with libc++/libc++abi/libunwind is just that its build is super complicated and I haven’t played around with it much, so I never included compiler-rt as part of the modernizations efforts I pushed forward. IMO, it would make sense for all the runtimes to have exactly the same compiler requirements, for simplicity and uniformity.

Having different compiler requirements means that either the testing infrastructure is going to be more complicated than it is today (that’s over my dead body), or that the older compilers supported only by libunwind/compiler-rt are going to go basically untested as far as the main developers who work on those projects are concerned. That’s not a great place to be either – I’d rather support only recent compilers, but do it excellently.

34

We’re not building a toolchain in the sense you’re using the word. We’re building a rust toolchain, that may use (parts of) compiler-rt and libunwind as runtime libraries. We are not building a C/C++ toolchain, so we do not build and don’t want to build clang.

The “you’re building a clang toolchain and might as well use a bootstrapping build” argument holds for libcxx and libcxx only – it does not apply to the compiler-rt/libunwind runtimes, which may be used with other toolchains.

In any case, the original proposal here was about libcxx only, it made no mention of libunwind or compiler-rt. The only reason I did not speak out against this proposal is that it was firmly libcxx specific (where it is both very reasonable on technical grounds, and does not materially affect me). So if there is a desire to adopt the same policy for other runtimes like libunwind or compiler-rt, I think that needs to be a new proposal.

For what it’s worth, there are no actual problems with building libunwind on older toolchains, I just want to make sure that nobody is going to start intentionally breaking it (or much worse, compiler-rt) citing this thread as their justification.

35

You haven’t answered my question – why are you unable to use one of the more recent compilers supported by libc++/libc++abi/libunwind? We support the two last stable releases of Clang and the last stable release of AppleClang + GCC, which provides reasonable turn around time for people (1 year in the case of Clang).

There are serious downsides to not being consistent across the runtimes. It provides a false impression of support when various bits of libunwind would actually never be tested consistently.

36

I suspect some of the reasons to want to use older compilers might have to do with supporting older distros? There’s an extensive discussion about this here:

Caveat: I have no stake in either of these discussions, and I am probably missing something obvious.

37

Right. Most of the concerns that apply to toolchain requirements for LLVM itself also apply to non-libcxx runtimes, and the same rigor should be exercised when raising requirements there.

In the rust case, the main problem is that we’re producing toolchains for something like 50 different targets, most of them cross-compiled, some with rather exotic cross-compilation setups. It’s not that getting a newer toolchain is impossible, it’s that updating so many different cross-compilation toolchains is a major amount of effort, and often runs into complications.

As a data point, the last tagged crosstool-ng release 1.24.0, which rust uses for many targets, has GCC 8.3 as the newest version – that’s quite recent and would even satisfy the new LLVM toolchain requirements from the linked RFC, but falls quite short of the GCC 11 requirement for libcxx.