RFC: Stand-alone build support

LLVM Project
22

Moving googletest out into the third-party directory has been suggested before. As a distro maintainer I would prefer having the option to use vanilla googletest, but I’m fine with either approach.

1 Like
23

That would be equally good and help a lot!

24

It seems likely we could figure out some way to do that by creating an llvm-googletest.h header in llvm, instead of patching googletest itself. (And also, after we increment the minimum C++ version to C++17, we may be able to switch to replace StringRef with std::string_view, removing that particular issue entirely).

1 Like
25

Hmmm there appears to be more to it (local patches to googletest) than that.
I haven’t identified the exact upstream revision used for the last merge, but it’s clear that we’ve done some things like add IWYU directives, and some pragmas to turn off -Wdeprecated-copy for selected bits. There are a few other isolated changes that I haven’t bothered to investigate.
The extensions I alluded to before probably can be done in a wrapper header, although I haven’t looked at that deeply either.

But, getting back to the point of this thread, the question is really whether doing something about googletest should be a blocker for standalone builds.

26

Polly also has a standalone build-mode, but I haven’t tested it for a long time. I tend towards removing it. I am also afraid that users might use mismatching version of Polly and LLVM (including downstream patches done by eg Debian), resulting in nonreproducible bugs.

27

@tstellar would you be willing to accept a backport for such a patch to release/14.x?

28

Yes, I think that would be worth backporting.

29

I opened https://github.com/llvm/llvm-project/issues/54716 to track this.

30

I think getting clarity around stand-alone builds is important, so I would not like to see this blocked while we wait to do something about googletest.

31

I think clang is the only consumer of LLVM’s googletest, outside of llvm itself, so while losing clang’s unittests is not great in a standalone build, it’s probably okay for that not to be a blocker.

32

lld also uses googletest. But the unittests already don’t work for stand-alone builds that don’t have access to the llvm source code.

33

Polly uses googletest as well.

llvm-test-suite has its own copy of googletest

1 Like
34

Based on the responses, it appears there is consensus for this, so I’m going to move forward with implementing this proposal. My goal is to order the changes from least to most impactful. Here are the changes I’m planning to make:

  1. Submit patches to clang, lld, and lldb to get stand-alone builds (as defined by this proposal) working and add documentation.
  2. Add a buildbot for testing stand-alone builds.
  3. Remove usage of llvm-config from clang, lld, and lldb CMake.
  4. Look into improving our googletest usage, and drop support for using LLVM_MAIN_SRC_DIR in stand-alone builds.

I’ll keep this topic updated with patches as I submit them.

2 Likes
LLVM Weekly - #433, April 18th 2022
35

I’ve posted the clang patches:

Build Fix: ⚙ D123967 Disable update_cc_test_checks.py tests in stand-alone builds
Documentation: ⚙ D123968 docs: Add instructions for stand-alone builds of clang

36

lld patches:

Build Fix: ⚙ D124315 lld: Bundle compact_unwind_encoding.h in lld to fix stand-alone builds
Documentation Update: ⚙ D124405 docs: add instructions for stand-alone builds of lld

The lld patch is more invasive than some of the others, because it relies on a header in libunwind.

37

@tstellar There’s a question of whether we should support sparse checkouts or whether we should always require full LLVM checkout that was raised on ⚙ D124315 lld: Bundle compact_unwind_encoding.h in lld to fix stand-alone builds. If we decide on the latter, that is requiring full LLVM checkout, a related question I have is whether we need to support building individual projects as standalone CMake builds?

The direction I have in mind is the following. Today it’s possible to do:

cmake ../clang && ninja clang

to build just Clang against prebuilt libLLVM.

I’d like to propose that we not only always require a full LLVM checkout, but also deprecate standalone builds of individual projects, and instead extend LLVM_ENABLE_PROJECTS.

Today, you can already do the following:

cmake ../llvm -DLLVM_ENABLE_PROJECTS=clang && ninja

but doing so also builds LLVM itself because LLVM_ENABLE_PROJECTS handling is done inside the LLVM build and thus always pull in the rest of LLVM:

I’d like to change that and instead lift the handling of LLVM_ENABLE_PROJECTS to the root, so the example above would only build Clang against the prebuilt libLLVM:

cmake .. -DLLVM_ENABLE_PROJECTS=clang && ninja

If you wanted to build Clang and LLVM together, you’d have to use:

cmake .. -DLLVM_ENABLE_PROJECTS=clang;llvm && ninja

This would let us remove the standalone build handling logic which is currently duplicated in all projects and introduces maintenance overhead.

This matches the model we use for runtimes. For example, with the following:

cmake ../runtimes -DLLVM_ENABLE_RUNTIMES=libcxx && ninja

you will only build libc++ and no other runtimes, but the common CMake configuration can be shared across all runtimes. This is the model I’d like to use for projects as well.

Would the model I described above work for your use cases?

2 Likes
38

This seems like a good idea to me, but would we really need to require a full llvm-project checkout in order to make this work? The reason I don’t like requiring the whole llvm-project checkout is it’s hard to prove that when you build clang for instance, you aren’t actually referencing some header or other file in the llvm/ directory. For our usecases, we like to know what code we are building so we can easily reproduce the builds and also so we can ensure export and license compliance when we ship products.

1 Like
39

You’re right, we would only need the top-level cmake directory and CMakeLists.txt file which could be packaged separately.

40

I worked on LLDB standalone builds a while ago. Back then it was also used to allow development with Xcode, which was overloaded with the entire LLVM/Clang/LLDB.

Streamlining the process across subprojects would be really good I think. Here’s a few pointers from the top of my head.

The LLDB standalone bot is here and apparently it’s still working: LLDB Standalone [Jenkins]
Build instructions and some context is here and apparently didn’t change much: Building - 🐛 LLDB

41

Just for your information. I’ve been working on getting a standalone buildbot worker up an running in the staging environment of buildbot. I’ve called the worker and the builder standalone-build-x86_64.

Here’s the config for the builder: https://github.com/llvm/llvm-zorg/blob/2b96a257f15dce470fe5b63d8ec46947d03e4e36/buildbot/osuosl/master/config/builders.py#L2427

It currently builds llvm and then clang in standalone mode. Once the first build is through I’m going to add lld and then maybe lldb depending if it builds properly. I still have to add check steps. So far we build each sub-project and then install it for later consumption by the next sub-project.

Here’s the more or less self- explanatory annotated build script (thanks @gkistanova for pointing me to annotated build scripts):

https://github.com/llvm/llvm-zorg/blob/main/zorg/buildbot/builders/annotated/standalone-build.sh

The buildbot worker is run as a container using this setup: https://github.com/kwk/llvm-build-standalone.

This is nothing more than a start to put the pieces in place and see them work.