Hey Ben,
Thank you for providing this feedback. I’m going to lay out some ideas that I have inline below.
I’m great with moving the runtimes into their own directory and making cmake modules to standardize an interface between the LLVM build process and the runtime build process. I would like to ask for more though.
I find working with the runtimes a bit frustrating at times, and I think a lot of that frustration stems from the second-class nature of the runtime build process. The “best” way to build just the runtimes today is to clone them in the llvm tree, pass a very long cmake line, then run a very specific make / ninja line (i.e. ninja cxx, ninja check-libcxxabi). This is even the case if I don’t want to use the freshly generated compiler. Here are some specific nuisances that running in-tree has caused me:
Agree on all points. It might be useful to provide an alternate top-level CMake file for runtime projects. There is some complication because the runtimes do use CMake modules from LLVM, so you’ll need either an LLVM checkout or a built & installed LLVM. My general hope in formalizing a runtimes directory is to start treating runtimes as first-class members of the LLVM project. Admittedly I’m coming from a different perspective than you, so I’m tackling the "LLVM & Clang developer who also wants runtimes” side of the problem. I am, however, sympathetic to your side too.
- I routinely need to pull a new LLVM for new cmake files, even though I am not building LLVM.
Some of this may be avoidable through restructuring our CMake modules. I think we should discuss this separately from the changes I’m asking for, but my general idea is it might be reasonable to create a separate LLVM repository that stores common CMake modules. If we did that it would require that everyone building LLVM would need to have those modules. We may be able to hide that complexity using CMake’s ExternalProject stuff. I’ll take it as a line item to look into that.
- I don’t get to use “standard” commands to build and test the components I work with. I want to just run make / ninja with no arguments, but instead, I need to build specific targets.
This is the kind of thing I see as potentially solvable with a separate top-level CMake file for runtimes. That said, in general I’ve been moving toward adding more and more explicit targets into CMake. As a result of that my natural workflow is involving less and less running general commands and more and more running specific “ninja ” and “ninja check-llvm-”.
- Choices made for LLVM’s build end up affecting mine, and I don’t have a good way to change them. For example, I wanted to perform some libcxx builds without -Wl,-z,defs. I had to do some wacky string filtering to remove it from the compile line after the fact, instead of preventing it from getting added in the first place. There wasn’t a flag available on the LLVM side to disable it at configure time.
LLVM’s flags impacting libcxx is fixed by my runtimes proposal. In fact, that’s part of the point. Bleeding options from LLVM & Clang builds into runtime libraries is not cool. It causes lots of problems on Darwin, so we’re sensitive to this.
- To get cmake to work, I have to set HAVE_CXX_ATOMICS_WITHOUT_LIB, even though I have no intention of building LLVM. I then get to set LIBCXX_HAVE_CXX_ATOMICS_WITHOUT_LIB too, because reasons.
This is bad. I’m curious why you need to set those ever. Have you diagnosed this? For you to need to set that it means the host toolchain isn’t properly passing the CMake checks.
- Multi-libs require multiple independent build directories, with all the associated cmake overhead.
I strongly believe that for building multi-lib or more generally when building for multiple targets you want multiple build directories, and in particular the multiple-cmake invocations. I believe this because you want the checks to be relevant for the target, and the only way to do that is to run the checks once per target.
That said, I also strongly believe that for any user of our projects we should find a way to have a single simple CMake invocation that gets the end result that you want. I don’t believe these are mutually exclusive goals.
One of the development steps of the new runtime directory will be supporting specifying multiple targets to build the runtimes for, and having CMake construct the appropriate number of build directories and manage building them all through a single top-level configuration and build directory. If you’re skeptical about how doable this is I’d encourage you to look at this bot → http://lab.llvm.org:8011/builders/clang-3stage-ubuntu.
That bot does a full 3-stage clang build from a single CMake invocation:
cmake -C …/llvm.src/tools/clang/cmake/caches/3-stage.cmake -GNinja -DLLVM_TARGETS_TO_BUILD=all -DLLVM_BINUTILS_INCDIR=/opt/binutils/include …/llvm.src
So why not run out of tree instead you may ask?
- No lit tests or lit utilities (FileCheck, not, etc…)
I don’t know if the runtimes you’re building are setup for this or not, but you can get out-of-tree tests working if you have an LLVM installation on the system or a build directory that you can point at. Compiler-RT does this. It isn’t ideal but it is workable.
We should have a better solution.
- Even more difficult to manage dependencies between libcxxabi and libcxx
Yep. That sucks.
So some things I would like to see…
- Standalone runtime builds should use the “normal” build interfaces (bare make, make all, make check, make install. s/make/ninja as desired).
I think this is doable, but I’m hesitant to rope it in with what I’m trying to do here. Nothing I want to do would prevent this or make it any harder than it already is.
- For in-tree builds, LLVM would use the new cmake ExternalProject feature. This way LLVM’s in-tree support could be implemented in terms of the runtime’s “normal” build interfaces. LLVM may also define a refinement of that interface to provide extra LLVM information on top.
- For example, maybe all llvm runtime projects get passed something like LLVM_RUNTIME_LIBCXXABI_PRESENT and LLVM_RUNTIME_LIBCXXABI_PATH, and those projects can act on that or not.
Yes, there will need to be a mechanism for communicating project dependencies between runtimes.
- Developers using standalone builds can use the same “LLVM build” interface as the in-tree builds use.
Yes. I’ve actually been having some hallway conversations about how to standardize the build interface a bit more cleanly. Some of this will depend on a per-project basis, but I think compiler-rt does some of this right today.
Specifically if an LLVM build tree is available it builds as if it were in-tree and being distributed with that build. If the LLVM build tree isn’t available, it builds in a more standard *nix format that would be compatible with non-LLVM toolchains. I may be wrong, but I think that is probably the right behavior for all our runtimes.
- Break out testing infrastructure to a common repo, so that the runtimes can have access to the testing “banana” without dragging along the LLVM "gorilla”.
I’m hesitant to suggest more and more repos because I think there are some challenges and additional burdens with that. I do understand the benefit of what you’re asking for here, and I think it is worth considering. I think there is an argument for splitting out the LLVM testing infrastructure, as well as an argument for splitting out the LLVM build infrastructure.
In both cases I think those changes are larger than what I’m proposing, but worth considering.
-Chris
Obligatory troll: Maybe we should move to github and change the whole repo structure in the process?
