Tips for incremental building

Beginners
1

As someone who is new to building the llvm-project source tree, I wanted to gather a list of tips for fast/efficient incremental developer builds (edit one file, compile, test, repeat). My most general question is, how do “power user” developers of llvm build the project? I am building both on Mac OX (xcode command line tools) and Linux for the respective architectures.

I have two specific questions below:

Speeding up incremental builds

What I have been doing so far is to start a fresh clean build with cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang-tools-extra;clang" -DCMAKE_BUILD_TYPE=Debug ../llvm, which then tends to build around 2.9K individual targets when I build the clang-tidy target:

$ ninja clang-tidy
[1/2889] ...

This takes a while of course, but once I’m done, then I can edit a single file and the build will incrementally only build the single object file, library archive, then executable. Linking the executable can take a minute or two even with the lld linker (noting I am doing a Debug build, which is going to lead to slower link times than Release). Are there any tips for speeding this incremental step up? I saw BUILD_SHARED_LIBS, although I haven’t tried this yet.

Rebasing against upstream rebuilds the entire project

I noticed that, after performing a full build against the main branch, if I pull the latest commits from upstream (about a day or two’s worth of commits), then run ninja, the project reconfigures, then builds what appears to be the entire project as if it were a clean build. These are my observations after having pulled changes three times (again, a day or two’s worth of commits), so it’s a small sample set, and perhaps I was just unlucky in which file(s) were changed. In general, does pulling down latest commits tend to trigger full project rebuilds? Any tips that I should be aware in this space?

1 Like
2

Use SSD, if possible.
BUILD_SHARED_LIBS=ON should improve link time significantly.
LLVM_CCACHE_BUILD=ON is a must if you switch between git branches often. Does not hurt otherwise, so I’d suggest using it. Make sure you configured ccache, the default 5GB cache size might not be enough, especially with static library builds.
LLVM_OPTIMIZED_TABLEGEN=ON helps with speeding-up debug build by using release version of llvm-tblgen.
These three are the most useful ones.
There are also options to disable certain subprojects / targets. See Building LLVM with CMake — LLVM 18.0.0git documentation for more or less complete list.
I also use CMAKE_C_COMPILER=clang, CMAKE_CXX_COMPILER=clang++ and LLVM_ENABLE_LLD=ON, but I can’t say for sure if this speeds up or slows down the build.

Unfortunately, yes. Commits that change core functionality are not rare. I’d suggest to rebase only if it is necessary (e.g. some critical bug has been fixed).

2 Likes
3

Thanks - these tips make sense.

Thanks for clarifying this. I suppose there’s little reason to pick up new changes in many cases, especially if I am developing clang-tidy or another tool. This reminds me of RFC: Stand-alone build support, and quick Googling finds projects like GitHub - OleksandrKvl/clang-tidy-standalone. It doesn’t look like workflows like this are “productionized”/officially supported in the LLVM source tree (please correct me if these kinds of workflows are supported). In any case, keeping a build directory around long term and basing feature branch work off of a tag, such as the most recent release of LLVM, is a close approximation.

Ah, thanks for pointing this option out. I’m also a fan of sccache and its remote caching support. I was able to get this to work successfully with -DCMAKE_CXX_COMPILER=$(realpath ../g++_wrapper) and injecting sccache in the wrapper script. I wonder if anyone would be interested in seeing LLVM_SCCACHE_BUILD supported?

1 Like
4

IMHO the bespoke ccache option should probably be removed now that CMake has support for launchers with CMAKE_C/CXX_COMPILER_LAUNCHER

4 Likes
5

Runtime libraries (compiler-rt, libcxx, a few others) support out-of-tree build. They can also be built using system compiler or even a cross-compiler. Other sub-projects like clang are tightly coupled to LLVM core libraries and thus should be always in-sync with them (C++ API is not stable).

Just in case, I would advise against basing your feature branch on release tags. It will be pain to rebase your feature branch onto newer major release.

1 Like
6

What I do:

  • Use ccache as mentioned before.
  • Do a release build with assertions enabled. This makes debugging a bit harder, but for me the link time improvements are well worth it.
  • For incremental changes, I don’t do a full ninja check-llvm-foo-bar but instead (for example) ninja lld and then I manually invoke lit (you can get the command with ninja -v check-foo-bar). Once your test of interest passes, you can run the full test suite to be sure everything still works.
  • Use ld.lld instead of ld.bfd, but you’re already doing that.
  • Use fast hardware. …that probably isn’t a great answer but it certainly helps a lot.

I don’t build with shared libs, I should probably start doing that.

2 Likes
7

Good point - my build works perfectly with sccache (using cmake -DCMAKE_CXX_COMPILER_LAUNCHER=sccache ...).

When I switched over to a shared libs build on Linux with GCC as my toolchain, linking is significantly faster.

Thanks all for the tips - my experience has been improved since (manually invoking lit was a good tip as well).

1 Like
8

For what it’s worth, buying a refurbished datacenter server made building Clang+LLVM not painful for me; I’d say “fast hardware” is a great answer – as long as that’s feasible.

1 Like