This message is both a heads up and a request for comments on an upcoming change we want to make to the way we build libc++/libc++abi/libunwind.
For a long time now, those three projects have supported various ways to be built:
The Project build, where one would use /llvm as the root of the CMake invocation and pass LLVM_ENABLE_PROJECTS.
The Standalone build, where one would perform one CMake invocation per project being built, and fill in various CMake variables to tell each project where to find bits of the other projects.
The “Runtimes” or “Bootstrapping” build, where one would use /llvm as the root of the CMake invocation and pass LLVM_ENABLE_RUNTIMES. This would result in Clang being built, and then the runtimes being built with that fresh Clang.
The “Unified Standalone” build, where one would use /libcxx/utils/ci/runtimes as the root of the CMake invocation and pass LLVM_ENABLE_PROJECTS. This was introduced to work around the fact that the Project build doesn’t work on several embedded platforms, and I think it is only used at Apple.
Through the years, this has gotten rather confusing and difficult to support. We currently support all of those on a best-effort basis and we do have build bots checking each of them, however the complexity is often getting in the way of providing proper support and improving the build system of those projects. We’ve been trying to migrate to a simpler way to build the runtimes that works for all use cases (notably embedded platforms) for a long time, and we recently got to a point where this is possible (thanks mainly to @phosek and @mstorsjo). To summarize what we’re aiming for, we’ll now have two ways of building libc++/lib++abi/libunwind:
The “Default” build, where the CMake invocation is rooted at /runtimes and one uses LLVM_ENABLE_RUNTIMES.
The “Bootstrapping” build, where the CMake invocation is rooted at /llvm and one uses LLVM_ENABLE_RUNTIMES (this is the current “Bootstrapping” build unchanged).
If you have been using the “Project”, “Standalone” or “Unified Standalone” builds, you should be moving to the new “Default” build. This should be really easy to achieve, basically you can just change LLVM_ENABLE_PROJECTS to LLVM_ENABLE_RUNTIMES, perform a single invocation of CMake instead of multiple invocations, and pass all the LIBCXX_FOO, LIBCXXABI_FOO and LIBUNWIND_FOO options to that single CMake invocation. See the release note in the PR linked below for more details.
We are aiming to deprecate all the old ways to build as soon as possible, and to keep supporting them until after the LLVM 14 release, at which point we would like to remove support for those and get started with making improvements. The review at https://reviews.llvm.org/D111356 enshrines this deprecation in the documentation. For technical discussion on this proposed change, please comment on that review to keep everything in one place.
Cheers,
Louis
Questions you might be asking yourself:
Q: Why doesn’t the Project build work for embedded platforms?
What should the buildbots do after this change? Should they all use
the bootstrap build? I am worried that this would reduces the test
coverage since it would only build the runtimes with the top-of-tree
clang. As by [1], at least libc++ supports also compilation with gcc
and AppleClang.
Name suggestion for the "default" build: "single target" build. Since
only the bootstrap build supports multiple targets with
LLVM_RUNTIME_TARGETS.
What should the buildbots do after this change? Should they all use
the bootstrap build? I am worried that this would reduces the test
coverage since it would only build the runtimes with the top-of-tree
clang. As by [1], at least libc++ supports also compilation with gcc
and AppleClang.
The build bots that wish to build libc++ with a supported non-clang-trunk compiler
can switch to the “default build”, which is basically the equivalent of the Project
build we had before. The procedure for doing that is explained in the release notes
here: https://libcxx.llvm.org/ReleaseNotes.html#build-system-changes.
Please notify me or someone in the libcxx Discord channel if you encounter issues
with making such a change. It should be straightforward, but I’d like to make sure
the transition is frictionless for everyone, so let me know if you encounter issues
and we’ll try to solve them.
Name suggestion for the “default” build: “single target” build. Since
only the bootstrap build supports multiple targets with
LLVM_RUNTIME_TARGETS.
Interesting suggestion. I’ll definitely keep that in mind. I’ll open up a review to officialize
the name of the new “default” build and pitch that suggestion there, thanks!
Which buildbots will test the "default build" for gcc/AppleClang/Clang
11/Clang 12? Individual buildbot maintainers may be interested in
keeping their buildbot green, not necessarily to ensure coverage.
The build bots that wish to build libc++ with a supported non-clang-trunk compiler
can switch to the “default build”, which is basically the equivalent of the Project
build we had before.
Which buildbots will test the “default build” for gcc/AppleClang/Clang
11/Clang 12? Individual buildbot maintainers may be interested in
keeping their buildbot green, not necessarily to ensure coverage.
That pipeline contains all the configurations that we support officially. If someone has a build bot and
they think they are adding coverage, we’d like to know it so we can look into migrating it to BuildKite
and owning it ourselves.
Otherwise, they should just make whatever change is needed to keep their bot green. In most cases I
would expect the easiest thing to do here would be to use the Bootstrapping build so as to always build
against the latest Clang, however they can also use the “Default build” with a suitably-recent system
compiler if they want to avoid building all of Clang on their bot.
I didn't know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, ...)
Do libcxx/abi, clang code reviews on phabricator trigger these buildkite builds similar to some of the other monorepo content?
Yes. There is both a rolling build (every 4 hours I think) building main, and also the full CI pipeline runs every time you upload a review on Phabricator that touches libc++, libc++abi or libunwind.
It’s been working like that for roughly one year now and it’s the best thing ever, we ship way fewer bugs and our development velocity has increased a lot.
I didn’t know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, …)
For libc++/libc++abi/libunwind, it’s actually quite simple. There’s exactly one place to look, and it’s BuildKite. And it runs systematically on all changes.
Buildbot: I removed all the libc++/libc++abi/libunwind buildbots over the past year.
Green Dragon: Ditto, I removed everything and we have Mac builders in BuildKite.
GitHubActions: Nobody’s using that yet AFAICT.
I will actually have a talk at the upcoming Dev Meeting where I will advocate for all projects moving to the same model, since we’ve had so many incredible benefits from doing it in libc++ & friends.
I didn’t know. Good that there is systematic testing, but I am a bit
concerned about the fracturing of the CI infrastructure (Buildbot,
Green dragon, GitHub actions, Buildkite, …)
For libc++/libc++abi/libunwind, it’s actually quite simple. There’s exactly one place to look, and it’s BuildKite. And it runs systematically on all changes.
Did you find a solution for having blamelist and blame emails on BuildKite?
Improving the Buildkite UI to have a good log scrubbing to the level of what we have with buildbot is doable, but I didn’t really see how to fix the above.
(I’m doing the opposite move right now: leaving BuildKite for MLIR CI to adopt buildbot).
Seems to be used for the release branch. Similarly, I didn't know that
we were using BuildKite for anything else than pre-merge checks before
you mentioned it.
No, but I haven’t tried. My experience so far has been that I’ve never had the need for a blamelist. Since we test everything pre-commit, you always know when one of your patch fails because you see it before it’s been committed, in Phabricator. Actual breakage on main is incredibly rare nowadays and it pretty much only happens when someone (usually myself) is too keen on checking something in before CI is finished (oops!). There’s never a lot of suspicious commits on main since the last build, so this isn’t a problem we’ve encountered.
I guess your experience can vary based on the commit traffic to the project and how diligent people are at testing everything pre-commit, of course.
as the commit traffic increases, you can’t necessarily test everything in sequence (Rust does, but their maintainers also have to manually bundle pull-request together: they are limited to ~7 merges per day), so concurrent merges can interact with each other and break master.
as the matrix of configurations increase, it does not seems possible to just test everything (HW: PPC, SystemZ, X86, Arm, Arm64, Risc TIMES the OSes Mac, Windows, Linux, FreeBSD TIMES the host compilers gcc 5/6/7/…, clang, MSVC, … TIMES sanitizers build), plus some things like the multi-stages bootstrap and such.
So in practice yeah I believe blamelist and notification are a must for LLVM and the monorepo, libc++ may be “too small” to be representative of the scale there.
This sounds like a huge simplification, thank you!
With this change, will we still be able to use LLVM_ENABLE_PROJECTS to build libc++, libc++abi and libunwind? IIUC, you are disabling this option entirely, right? Will CMake warn us about the incorrect/deprecated usage? Should the LLVM documentation be updated [1]?
This sounds like a huge simplification, thank you!
With this change, will we still be able to use LLVM_ENABLE_PROJECTS to
build libc++, libc++abi and libunwind? IIUC, you are disabling this
option entirely, right? Will CMake warn us about the
incorrect/deprecated usage? Should the LLVM documentation be updated [1]?
Hi,
The goal is to deprecate and then entirely remove building the runtimes with LLVM_ENABLE_PROJECTS. Libc++'s own documentation has already been updated, but I will update the documentation you linked to and add a warning in the CMake – thanks for the suggestion!
This sounds like a huge simplification, thank you!
With this change, will we still be able to use LLVM_ENABLE_PROJECTS to
build libc++, libc++abi and libunwind? IIUC, you are disabling this
option entirely, right? Will CMake warn us about the
incorrect/deprecated usage? Should the LLVM documentation be updated [1]?
Hi,
The goal is to deprecate and then entirely remove building the runtimes with LLVM_ENABLE_PROJECTS.
Glad to hear that’s on the plan - looking forward to removing the ambiguity/traps.
I’m looking to find and document the best ways to use clang thread-safety features (capability analysis), especially for some common patterns of access such as a single reader/writer thread with multiple reader threads, and also to document how to use it for thread-locked access checking – there’s a single example involving a ThreadRole.h file to set up uses of capabilities for tracking threads. There’s a brief mention of it on the paper, and also a longer example at . However, that’s from 2014, and it doesn’t cover a lot of possible uses.
One way to possibly handle the reader/writer vs readers case (where reads on the writing thread don’t need to lock) would be to be able to say “guarded by this or that”, in this case something like GUARDED_BY(mMutex, MainThread) (GUARDED_BY(mMutex || MainThread) ??).
I believe it’s mostly unmaintained at this point. +Delesley Hutchins (as the primary developer on this originally, or at least the last maintainer I’m aware of) in case he’s got any pointers to current state/interested parties.
I believe it's mostly unmaintained at this point. +Delesley Hutchins (as the primary developer on this originally, or at least the last maintainer I'm aware of) in case he's got any pointers to current state/interested parties.
Aaron Puchert (CCed) has been doing a lot of good work in this area,
and I still continue to help review patches with the functionality.
Delesley has also been very helpful with review work when we need to
reach out for his expertise, so this is still being actively
maintained.
All of our thread safety analysis documentation lives at:
If you are interested in improving this documentation, I'm sure
there's a few of us who would be happy to help review any improvements
you want to make!
Thanks for letting me know. (I barely follow the list.)
One way to possibly handle the reader/writer vs readers case (where reads on the writing thread don't need to lock) would be to be able to say "guarded by this or that", in this case something like GUARDED_BY(mMutex, MainThread) (GUARDED_BY(mMutex || MainThread) ??).
There has been some work on logical expressions in capability attributes in rG7c192b452fa2, but I believe it's not functional yet. I thought about the semantics of this a bit, but don't have a good understanding yet.
If you want accesses to a resource to be protected, it's in general not sufficient to have one of a set of capabilities. There can only be one. What happens in this scenario is that there is a certain period of time where the resource is exclusive to the main thread, and another where it's only accessible via mutex (even if the main thread were to access it). So maybe logical expressions aren't the right way to understand this situation, but rather there should be different types, one that has GUARDED_BY(MainThread) and another that has GUARDED_BY(mMutex), and at some point we convert between them. In some sense the protection regime changes throughout the lifetime, and for static analysis that means we need a new (static) type.