[RFC] Use pre-compiled headers to speed up LLVM build by ~1.5-2x

[aengelke]

Building LLVM is slow, this is a frequent complaint, especially from people with weaker hardware. Build times are dominated by the C++ front-end, (repeatedly) parsing headers is the most time consuming part. C++ module builds don’t really help (for me), are fragile, and kill parallelism.

Therefore, I propose to extensively, but optionally, use pre-compiled headers (rough draft PR) for frequently used headers (i.e., C++ stdlib + Support, IR, CodeGen, gtest). This can substantially reduce build times of LLVM by ~1.5-2x, front-end time by ~3x (see breakdown below). stage2-clang build times improve by ~40% on c-t-t. On my laptop (M2 MacBook Air, 4+4 cores), a dual-target LLVM build now just takes 6-7 minutes (incl. tools), unit tests add another 2.5 min.

Note that pre-compiled headers are already used in-tree by Flang for some libraries to speed up compilation (e.g. here). This proposal implements a much more extensive approach, where almost all libraries would use PCHs. Further build time improvements are possible by using more different PCHs for other libraries like clangAST and dependents.

Downsides: using PCHs has two caveats:

1. Regular build fails but PCH build succeeds: PCH masks missing includes.
2. Regular build succeeds but PCH build fails: much more “used” includes can cause naming collisions (e.g. llvm::Reloc from llvm/Support/CodeGen.h vs. lld::macho::Reloc from lld/MachO/Relocations.h; where e.g. lld/MachO/InputSections.cpp uses both namespaces) and (rarely) ambiguities due to more available implicit conversions.

We’d therefore need CI (at least post-commit) that regularly tests PCH and non-PCH builds to make sure that both builds work.

General Questions

• Does this extensive PCH use has a reasonable chance of getting merged upstream? (If not, I’d not spend more time in polishing this into a mergeable state.)
• Enable by default vs. not?
• Flang already uses PCH – what are experiences worth noting?

---

Details: Implementation (+Technical Questions)

The current patch builds four pre-compiled headers. This is “somewhat arbitrary” in that I simply selected the headers that show up with long parse times accumulated over all CUs.

• LLVMSupport, which includes all C++ standard headers and frequently used headers from llvm/Support and llvm/ADT.
• LLVMCore, which extends LLVMSupport with frequently used headers from llvm/IR.
• LLVMCodeGen, which extends LLVMCore with frequently used headers from llvm/CodeGen.
• clangAST, which extends LLVMSupport with some headers from clang/AST.
• llvm_gtest, which extends LLVMSupport with gtest/gtest.h.

Libraries that depend on CodeGen reuse the LLVMCodeGen PCH, depend on Core but not CodeGen reuse the LLVMCore PCH, depend on Support but not Core reuse the LLVMSupport PCH.

I currently put the header list in include/llvm//pch.h. Not the best place (we might not want them installed), but lib/ is also not ideal, as e.g. IR/pch.h would include “../Support/pch.h”, which also feels wrong. We could also keep them in CMakeLists.txt, but that’d make reuse of the list more awkwards (e.g., extending the list from LLVMSupport to LLVMCore, which should be a superset).

• Where to store header list for PCH?
• Add separate option to enable/disable vs. just rely on standard CMAKE_DISABLE_PRECOMPILE_HEADERS?
• Which CI runners use PCH vs. which don’t? (NB: ccache supports PCH, sccache apparently doesn’t.)
• How to not hard code the PCH selection in AddLLVM.cmake? (replaced with simple dependency chain length heuristic)
• There’s a minor perf regression in PCH builds, which I haven’t yet investigaged – any ideas why this could be? (I’d have expected the output to be nearly identical.) (I accidentally included iostream…)
• Two unittests (flang/unittests/Evaluate, clang/unittests/Interpreter/ExceptionTests) use exceptions and also enable RTTI. To build the llvm_gtest PCH without RTTI, I changed the llvm_gtest build to not forcefully enable RTTI. Are there expectable problems from building these two tests with -fexceptions -fno-rtti? (works for me) (Tangentially related: what is the reason why -fno-exceptions -funwind-tables show up in llvm-config --cxxflags? Using exceptions when not unwinding through LLVM should be fine, so only -fno-rtti should be there?)
• If we want PCH by default, on which platforms? There appear to be problems with the Flang PCHs on Windows.
• (Note to self: the Flang-specific parts here need to be moved to AddLLVM) (done)

Details: Data

Time breakdown (seconds; collected with -ftime-trace) with a X86+AArch64 -O1 build on a 48-core machine:

                                   main +PCH
--------------------------------- ----- ----
ExecuteCompiler                   11528 5653
Frontend                           9128 3388
  Source                           6326 1488
  PerformPendingInstantiations     2160 1124
  CodeGen Function                  287  329
Backend                            2333 2201
  Optimizer                        1539 1454
  CodeGenPasses                     786  741

wall time (48c/96t)                 139   86
CPU time (usr+sys)                11706 5919

cmake -DLLVM_TARGETS_TO_BUILD="X86;AArch64" -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_FLAGS_RELEASE=-O1 -DCMAKE_CXX_FLAGS_RELEASE=-O1 -G Ninja -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_LINK_LLVM_DYLIB=ON -DCMAKE_C_COMPILER=.../clang -DCMAKE_CXX_COMPILER=.../clang++ -DCMAKE_C_FLAGS="-ftime-trace -ftime-trace-granularity=100" -DCMAKE_CXX_FLAGS="-ftime-trace -ftime-trace-granularity=100" -DLLVM_USE_LINKER=lld -B llvm-build && ninja -C llvm-build

NB: due to the high parallelism on this machine, the length of the slowest compile units becomes increasingly relevant, e.g. SLPVectorize.cpp takes >40s alone. Also note: using TPDE-LLVM as back-end can replace the time in CodeGenPasses with ~10s, but this will continue to be my local setup.

File size of the precompiled headers; total build directory size grows from 961 MiB to 1224 MiB with the config from above:

 105M .../LLVMCodeGen.dir/cmake_pch.hxx.pch
  69M .../LLVMCore.dir/cmake_pch.hxx.pch
  39M .../LLVMSupport.dir/cmake_pch.hxx.pch
  43M .../llvm_gtest.dir/cmake_pch.hxx.pch

Alternatives

• Make Clang 3x faster. I see no fundamental reason why parsing C++ has to be this slow, but this is very unlikely to happen (I’m not going to write a new C++ parser and I believe Clang still has the trend of becoming slower over time).
• Restructure code so that much fewer includes are needed. Lot of effort, unlikely to happen. Might also only have limited effect, because several standard library headers are very slow to parse.
• Use unity/jumbo builds (several .cpp files are compiled together). This reduces the number of times headers are parsed, but also reduces parallelism, increases memory usage, and increases the cost of incremental builds. (thanks, @makslevental)
• Use C++20 modules. This would require substantial refactoring and is unlikely to be feasible for the near/medium future due to our toolchain requirements. (thanks, @h-vetinari)
• Use Clang header modules (cmake -DLLVM_ENABLE_MODULES=ON). This substantially reduces build parallelism, is incompatible with libstdc++, and appears to be rather fragile with unclear/varying build time benefits.
• Rewrite LLVM in a language that compiles faster… haha, just kidding.
• Do nothing.

[makslevental]

@aengelke

Alternatives

• Make Clang 3x faster. I see no fundamental reason why parsing C++ has to be this slow, but this is very unlikely to happen (I’m not going to write a new C++ parser and I believe Clang still has the trend of becoming slower over time).
• Restructure code so that much fewer includes are needed. Lot of effort, unlikely to happen. Might also only have limited effect, because several standard library headers are very slow to parse.
• Rewrite LLVM in a language that compiles faster… haha, just kidding.
• Do nothing

There’s actually a 5th alternative with requires a refactoring of about the same scale: support UNITY_BUILD. I’ve actually taken a shot at it (using the help of clang plugin) but ran out of steam without getting to a workable state (so I don’t have numbers) but I feel the speed up should be roughly similar? Although with unity build you lose parallelism across TUs .

Anyway I’m strongly +1 on anything the speeds up build time (specifically because it makes LLVM “more inclusive” for people who can’t afford hardware).

[rengolin]

@aengelke

Make Clang 3x faster. I see no fundamental reason why parsing C++ has to be this slow, but this is very unlikely to happen (I’m not going to write a new C++ parser and I believe Clang still has the trend of becoming slower over time).

The first line up there got me hoping you would!

@aengelke

Restructure code so that much fewer includes are needed. Lot of effort, unlikely to happen. Might also only have limited effect, because several standard library headers are very slow to parse.

We may have gotten sloppy recently, but this has always been a core design property of LLVM, with the local headers and pervasive use of static / anonymous namespace functionality.

Recent code-helpers (semantic or AI based) made this even less of an issue, as I do sometimes get reminders that a header is unused.

To improve this side of LLVM we’d probably have to rewrite a lot of interfaces, which would be pervasive across the entire code base and likely downstream projects, so unlikely to be worth doing.

@makslevental

I’ve actually taken a shot at it (using the help of clang plugin) but ran out of steam without getting to a workable state (so I don’t have numbers) but I feel the speed up should be roughly similar?

I wasn’t aware of this, but it does seem like more of a trade-off than a net-positive gain. IIUC, this ends up building fewer bigger files, which may be a problem for memory constrained platforms. And the lack of parallelism would be a hit for offline builders many big companies have (like Bazel remote builds).

But it doesn’t hurt to have the option. It could even be an option for buildbots in Zorg, so that different hardware can build faster for their own configurations.

@makslevental

Anyway I’m strongly +1 on anything the speeds up build time (specifically because it makes LLVM “more inclusive” for people who can’t afford hardware).

+100!

[h-vetinari]

@aengelke

C++ module builds don’t really help (for me), are fragile, and kill parallelism.

What do you mean by “C++ module build”? To leverage the benefits of C++20 modules, a lot of refactoring would have to take place first (technically possible once LLVM requires C++20, but practically speaking not likely for a while yet due to the state of toolchains).

@ChuanqiXu had a recent blog post on modules. Taking a very selective quote that’s relevant to this topic

Some have argued that C++20 Modules are nothing more than standardized Precompiled Headers (PCH) or standardized Clang Header Modules. This is incorrect. PCH or Clang Header Modules reduce compile times by avoiding repetitive preprocessing and parsing across different TUs.

C++20 Modules go a step further by also preventing repetitive optimization and compilation of the same declarations in the compiler backend. For many projects, backend optimization and code generation are the main sources of long compilation times.

Even if rejected (or rejected for the foreseeable future), the Alternatives section should list moving to C++20 modules as an option.

[aengelke]

@makslevental

There’s actually a 5th alternative with requires a refactoring of about the same scale: support UNITY_BUILD.

True, I forgot that. However, unity builds additionally have the disadvantage of slowing down incremental builds, which are the typical case during development.

@h-vetinari

What do you mean by “C++ module build”?

I simply tried -DLLVM_ENABLE_MODULES=ON and I admit that I’m not 100% sure what it does right now.

For many projects, backend optimization and code generation are the main sources of long compilation times

According to my measurements above, this is not the case for large parts LLVM. I don’t know how Clang implements C++ modules, but inline functions still would need to be visible in the LLVM IR for optimizations, so would still need to be optimized?

I’m going to add the two alternatives to the initial post, thanks for bringing them up.

[ChuanqiXu]

Modules in clang have two meanings: one for clang header modules and one for C++20 modules.

-DLLVM_ENABLE_MODULES=ON should enable for clang header modules.

And I am curious why not use -DLLVM_ENABLE_MODULES=ON? I feel clang header modules should be better than PCH.

[aengelke]

Thanks for the explanation.

@ChuanqiXu

And I am curious why not use -DLLVM_ENABLE_MODULES=ON? I feel clang header modules should be better than PCH.

• It kills parallelism: on the 96-thread machine, rarely more than 20 clang++ processes were running at the same time and most of the time, only very few processes were running in parallel.
• On a my laptop, I compared build performance of header modules and PCH a few months ago: build times (wall time) were not noticeably faster than a regular non-PCH build, probably due to the reduced parallelism.
• I ran into problems with incremental builds more than once, where the incremental build failed with spurious errors that were gone when rebuilding from scratch (might’ve been fixed in the mean time, I’d need to check).
• At least on 2b903df7, on which I tried this again just now, building fails with errors like the following, which I don’t know how to fix. Maybe this is a libstdc++ problem, but that is very popular among Linux distributions. (Ubuntu 25.04 system, libstdc++ 15-20250404, Clang built from main ~mid-December)

The reduced parallelism seems to be an inherent property, which is likely to reduce wall-time benefits on typical developer machines (and wall-time is much more important than CPU time). Apart from this, due to the issues listed above, it seems to me that the implementation is not yet mature enough for general use.

While building module 'LLVM_Transforms' imported from /home/engelke/llvm-project/llvm/lib/Transforms/Utils/MemoryTaggingSupport.cpp:13:
In file included from <module-includes>:91:
In file included from /home/engelke/llvm-project/llvm/include/llvm/Transforms/Scalar.h:19:
In file included from /usr/lib/gcc/x86_64-linux-gnu/15/../../../../include/c++/15/functional:50:
/usr/lib/gcc/x86_64-linux-gnu/15/../../../../include/x86_64-linux-gnu/c++/15/bits/c++config.h:349:15: error: redefinition of '__terminate'
  349 |   inline void __terminate() _GLIBCXX_USE_NOEXCEPT
      |               ^
/usr/lib/gcc/x86_64-linux-gnu/15/../../../../include/x86_64-linux-gnu/c++/15/bits/c++config.h:349:15: note: previous definition is here
  349 |   inline void __terminate() _GLIBCXX_USE_NOEXCEPT
      |               ^

[ChuanqiXu]

This is inconsistency with my impression. I thought clang header modules are better PCH generally.

@Bigcheese @jansvoboda11 @zygoloid do you have any insights?

[philnik]

Clang Modules generally require all dependencies to be modular AFAIK, which libstdc++ isn’t. You might want to try compiling with libc++ instead, which has full support for clang modules.

[aengelke]

Thanks. With libc++, there’s more progress, but different errors (and warnings like -Wc++17-extensions).

In any case, I aborted after ~7 minutes at the progress of 740/2814 – while a regular build completes within less than 2.5 minutes on that machine (see above). (Most of the time less than 10 clang++ processes were running in parallel.)

(PS: at that point, the module.cache directory was already over 700 MiB, which does raise the question of the build directory size, which is also a concern (cf. discussion on whether to enable LINK_DYLIB by default).)

While building module 'LLVM_Object' imported from /home/engelke/llvm-project/llvm/lib/Object/Minidump.cpp:9:
In file included from <module-includes>:18:
/home/engelke/llvm-project/llvm/include/llvm/Object/IRObjectFile.h:16:2: fatal error: module 'LLVM_IR' is defined in both '/home/engelke/llvm-build-tmp7/module.cache/2DH669HT2GLJZ/LLVM_IR-RU9PNU9ENOZZ.pcm' and '/home/engelke/llvm-build-tmp7/module.cache
/2DH669HT2GLJZ/LLVM_IR-RU9PNU9ENOZZ.pcm'
   16 | #include "llvm/Bitcode/BitcodeReader.h"
      |  ^
/home/engelke/llvm-project/llvm/lib/Object/Minidump.cpp:9:10: fatal error: could not build module 'LLVM_Object'
    9 | #include "llvm/Object/Minidump.h"
      |  ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.

/home/engelke/llvm-project/llvm/lib/Object/DXContainer.cpp:9:2: fatal error: module 'LLVM_BinaryFormat' is defined in both '/home/engelke/llvm-build-tmp7/module.cache/2DH669HT2GLJZ/LLVM_BinaryFormat-RU9PNU9ENOZZ.pcm' and '/home/engelke/llvm-build-tmp7/module.cache/2DH669HT2GLJZ/LLVM_BinaryFormat-RU9PNU9ENOZZ.pcm'
    9 | #include "llvm/Object/DXContainer.h"
      |  ^
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /home/engelke/llvm-build/bin/clang++ -DGTEST_HAS_RTTI=0 -DLLVM_EXPORTS -D_DEBUG -D_GLIBCXX_ASSERTIONS -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/engelke/llvm-build-tmp7/lib/Obje
ct -I/home/engelke/llvm-project/llvm/lib/Object -I/home/engelke/llvm-build-tmp7/include -I/home/engelke/llvm-project/llvm/include -stdlib=libc++ -fPIC -fno-semantic-interposition -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availabili
ty-new -fmodules -fmodules-cache-path=/home/engelke/llvm-build-tmp7/module.cache -Xclang -fmodules-local-submodule-visibility -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wc++98-c
ompat-extra-semi -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wno-pass-failed -Wmisleading-indentation -Wctad-maybe-unsupported -fdiagnostics-colo
r -ffunction-sections -fdata-sections -O1 -std=c++17 -UNDEBUG -fno-exceptions -funwind-tables -fno-rtti -MD -MT lib/Object/CMakeFiles/LLVMObject.dir/DXContainer.cpp.o -MF lib/Object/CMakeFiles/LLVMObject.dir/DXContainer.cpp.o.d -o lib/Object/CMakeFiles/
LLVMObject.dir/DXContainer.cpp.o -c /home/engelke/llvm-project/llvm/lib/Object/DXContainer.cpp
1.      /home/engelke/llvm-project/llvm/lib/Object/DXContainer.cpp:9:2: current parser token 'include'
 #0 0x000072a4ac842c5b llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/home/engelke/llvm-build/bin/../lib/libLLVM.so.22.0git+0x842c5b)
 #1 0x000072a4ac8406ab llvm::sys::RunSignalHandlers() (/home/engelke/llvm-build/bin/../lib/libLLVM.so.22.0git+0x8406ab)
 #2 0x000072a4ac778af6 CrashRecoverySignalHandler(int) CrashRecoveryContext.cpp:0:0
 #3 0x000072a4ab8458d0 (/lib/x86_64-linux-gnu/libc.so.6+0x458d0)
 #4 0x000072a4b15acb2a clang::Redeclarable<clang::TagDecl>::DeclLink::getPrevious(clang::TagDecl const*) const (/home/engelke/llvm-build/bin/../lib/libclang-cpp.so.22.0git+0xbacb2a)
 #5 0x000072a4b17cae62 clang::CXXRecordDecl::addedMember(clang::Decl*) (/home/engelke/llvm-build/bin/../lib/libclang-cpp.so.22.0git+0xdcae62)
 #6 0x000072a4b2f9abdc clang::ASTReader::finishPendingActions() (/home/engelke/llvm-build/bin/../lib/libclang-cpp.so.22.0git+0x259abdc)
 #7 0x000072a4b2f9d1ca clang::ASTReader::FinishedDeserializing() (/home/engelke/llvm-build/bin/../lib/libclang-cpp.so.22.0git+0x259d1ca)

[zygoloid]

How many different module configurations are you seeing in your module cache? (Easy check: how many different PCM files for the LLVM support module? You can then investigate why they’re considered to be different using llvm-bcanalyzer -dump and looking at the header block, IIRC.) I remember there being problems in the past where bad cmake rules led to there being a lot of different sets of -D flags being unnecessarily passed to different builds.

I’ve seen the lack of parallelization stem from three things: one is pinch points in the build graph when building binaries that generate source code (like tablegen), another is the build system being really bad at scheduling in the presence of such pinch points (which cmake generator are you using?), and a third is lots of compiles blocking on the same module files (which would tend to suggest we’re spending too much time building module files, probably due to there being too many configurations).

I would recommend profiling the build using -ftime-trace (3.6. Performance Investigation — Clang 22.0.0git documentation) and looking at the resulting graph to see where you’re losing build performance.

[zygoloid]

I should add: at one point, -DLLVM_ENABLE_MODULES=ON did work and gave a substantial speedup to the build (around the same you’re seeing with PCH), so it seems like something has regressed here.

[aengelke]

@zygoloid

Easy check: how many different PCM files for the LLVM support module? You can then investigate why they’re considered to be different using llvm-bcanalyzer -dump and looking at the header block, IIRC.

I looked at this for a while and have zero clue on how to interpret the output. So all I can say is that there are 5 files names LLVM_Support_DataTypes_Src-RU9PNU9ENOZZ.pcm. I don’t think that -D flags are the problem, because they are not a problem with the PCH build?

@zygoloid

which cmake generator are you using?

Ninja

@zygoloid

third is lots of compiles blocking on the same module files (which would tend to suggest we’re spending too much time building module files, probably due to there being too many configurations).

I believe this is the reason – the other two possible causes would also appear in other build configurations. -ftime-trace doesn’t help with build system dependency problems.

[Tachyon]

One big problem that I know with PCHs is that they’re not supported by caching tools like ccache, sccache, fastbuild.

I don’t remember why this doesn’t work (something about includes in PCH?), but I always tend to choose a cacheable build rather than a somewhat faster build.

Edit: Just noticed the first post said ccache supports PCH, which I don’t agree with.

Further, it can’t detect changes in #defines in the source code because of how preprocessing works in combination with precompiled headers.

Not detecting changes doesn’t really sounds like it’s supported to me.

[zygoloid]

Hm. 5 configurations doesn’t sound like a huge number.

Looking at your previous message, I’m more worried by the errors/crashes you saw. I wonder if we are somehow loading the same PCM file more than once. I’ve given this a go myself using Clang 19 from Ubuntu, and I’m seeing different issues (though probably related): watching the module cache, I see lots of lock files for the same module getting created, then the module build completes, and then there’s a delay of many seconds (up to several minutes) before the lock files go away again. My build fails with errors like

/home/richardsmith/llvm-project/llvm/lib/DebugInfo/PDB/Native/NativeEnumTypes.cpp:9:2: fatal error: module file ‘/home/richardsmith/llvm-project/build-modules/module.cache/1K9BRQPF018UY/LLVM_DebugInfo_PDB-3DV09TEAYTM16.pcm’ is out of date and needs to be rebuilt

which I think likely indicates that we’re building the same PCM files multiple files and overwriting them during a build. Definitely looks like something in our implicit module building / caching system is broken (at least with Clang 19 – I’ve not yet tried building with Clang trunk).

[rnk]

The big downside of PCH and unity/jumbo builds (I will use “jumbo” terminology to avoid ambiguity with the popular game framework) is that code that builds with PCH/jumbo will often not build when compiled with traditional textual headers, and code that builds with textual includes can often break the PCH/jumbo build. The maintenance cost is significant, in my view. Neither of these configurations will remain viable without post-submit CI, and they will impose maintenance cost for those who don’t use them.

Clang header modules, on the other hand, should be more transparent, and while they have maintenance cost, the whole point is that the semantics are supposed to be as close to headers as feasibly possible.

I hate to make the perfect the enemy of the good, but it would be highly beneficial to the Clang project if we could eat our own dogfood and successfully deploy Clang header modules. It might be significantly more work than deploying PCH today, but it would be kind of a big deal. There’s been so much teeth gnashing about C++20 modules, it would be nice if we could demonstrate that the tech works. IIRC LLVM_ENABLE_MODULES still relied on implicit modules and lockfiles, not the new upfront clang-scan-deps approach, so maybe there’s something that can be done here.

---

As an impractical aside, back when Chromium dropped jumbo build support, I tried to pitch a project I called “semantics-preserving jumbo”, but nobody was excited to work on it and it kind of fizzled. The idea here was to use the same tech that Clang uses to control visibility of header submodules to “disable” visibility of all previously included headers until they were reincluded in the next intra-jumbo-CU-shard. So, the compiler processes file1.cpp, includes everything, hides all Decls, processes file2.cpp, and every #include just reactivates the relevant submodules. Internal linkage entities would also not be visible to lookup between cpp files, so you don’t have to give anonymous/static entities globally unique names.

[aengelke]

@Tachyon

Edit: Just noticed the first post said ccache supports PCH, which I don’t agree with.

Further, it can’t detect changes in #defines in the source code because of how preprocessing works in combination with precompiled headers.

Not detecting changes doesn’t really sounds like it’s supported to me.

I looked and tried to come up with a minimal example demonstrating the problem, but wasn’t successful so far – do you have an example where this causes false positives? In any case, we could disable PCH by default if ccache is enabled (similar to what was proposed in #141927, which was abandoned).

Flang does use PCH by default with ccache right now (#136856) and apart from #142449 (which wasn’t investigated but was gone with the new pre-merge CI) I’m not aware of issues this caused on non-Windows platforms. If we consider this to be a relevant concern, we should probably disable Flang PCH as well.

@rnk

The maintenance cost is significant, in my view. Neither of these configurations will remain viable without post-submit CI, and they will impose maintenance cost for those who don’t use them.

No doubt there is a cost and we’d definitely need post-commit CI for both build types, but the significance is, IMO, debatable. Missing headers (PCH succeeds, regular build fails) would be caught in pre-commit CI; so the other direction of naming collisions/ambiguities is more problematic. For the PCH set I used, in the entire LLVM/Clang/MLIR/Flang/LLD code base, there were only 5 problems in total. I’d also argue that our headers should be structured in a way that adding some includes doesn’t cause unrelated failures. I therefore see testing with more includes as beneficial for the project, which can be worth the cost.

(Waiting for slow builds is also a significant cost.)

@zygoloid

at least with Clang 19 – I’ve not yet tried building with Clang trunk

FWIW, I just tried building LLVM 18 with Clang 18 as well as LLVM main with Clang 18, and encountered some similar problems. Actually, I’d be interested in a configuration where header modules do work on a GNU/Linux system so that I at least can get rough performance numbers.

@rnk

it would be highly beneficial to the Clang project if we could eat our own dogfood and successfully deploy Clang header modules.

I agree in principle. But: this option exists since over 10 years and apparently has been broken on Linux systems in some ways for many months, maybe even years. It seems that almost nobody uses this option to build LLVM and also nobody seems to really care about it. Admittedly, I’d personally be also very hesitant to rely on an effectively unmaintained feature for productive work.

I’d expect getting this into a working state to be a non-trivial effort. Even if all issues were fixed in e.g. Clang 23, it’d take some time for the release to trickle down to distros, so it would take at least one year before any possible benefits see wider adoption.

And then there’s also the consideration of “developer education” about header modules. I – and I assume many others – have not a sufficient conceptual understanding of how header modules work and would probably be unable to properly fix build problems. There’s apparently a (rarely updated) modulemap file that specifies modules, so to add/move a header I need to learn a new DSL from this rather lengthy documentation? I would question that the maintenance costs of this is lower than of the costs of pre-compiled headers, for which at least the conceptual model is quite simple.

I’d personally favor PCH as a reasonably working solution that provides benefits right now also with not-the-latest and not-Clang toolchains. Should modules in whichever form be ready for wider adoption and prove to be sufficiently good alternative (at which point the ccache question will likely come up again), we can just remove PCH again.

[zygoloid]

I’ve done a bit more investigating. It looks to me like the in-memory module cache is responsible for the problem. What seems to be happening is:

• We try to load a module, and find one of its transitively imported modules is out of date.
• In the in-memory module cache, all the involved modules on the path to that transitively imported module are marked as ToBuild. This forces them to be treated as “out of date”, even if they get rebuilt by some other clang process and brought back up to date in the mean time.
• We start building one of the transitively imported modules. In the mean time, some other clang process builds another of those transitively imported modules, call it M.
• We then find we need M, and the in-memory module cache (incorrectly) says it’s ToBuild, so we rebuild it too, changing its mtime and causing all dependent modules to be out of date.
• This then forces more rebuilds due to those dependent modules being out of date.

In order for this to start going wrong, all we need is for any module that is widely imported to be considered to be out of date. And that happens due to tablegen and similar things running.

So… I applied this diff:

--- a/clang/lib/Serialization/InMemoryModuleCache.cpp
+++ b/clang/lib/Serialization/InMemoryModuleCache.cpp
@@ -66,7 +66,8 @@ bool InMemoryModuleCache::tryToDropPCM(llvm::StringRef Filename) {
   if (PCM.IsFinal)
     return true;
 
-  PCM.Buffer.reset();
+  PCMs.erase(Filename);
+  //PCM.Buffer.reset();
   return false;
 }
 

and now I get the following:

• Modules, clean build, warm cache: ninja clang 11736.42s user 1138.74s system 4997% cpu 4:17.65 total
• No modules: ninja clang 31212.51s user 1858.46s system 10594% cpu 5:12.15 total

That’s still not great in terms of wall time and parallelization, but it’s a 2.66x reduction in total CPU usage. And the modules build is wall-time faster than the non-modules build now, and doesn’t seem to be doing huge amounts of spurious module rebuilds any more. (The “clean build, warm cache” here means I did a “ninja -t clean” but did not delete the module cache. So this is not rebuilding the 3311 standard library PCM files used in this build (!) but will be rebuilding any module that depends on any generated file, which in LLVM is nearly all of them. This should roughly correspond to the rebuild time after something like git merge main.)

If someone wants to take the above diff and turn it into a PR (there’s a lot more stuff in the in-memory module cache that can be cleaned up as part of removing the ToBuild state; that diff is definitely not fit to land as-is), please be my guest; I don’t plan to.

But I agree with @rnk that we should investigate using clang-scan-deps instead of implicit modules here, as it’s probably not realistic to substantially improve the parallelization with the current approach. (Maybe we could look for a sequence of #includes that would all be translated to module imports, and if we see that the first one is being built by another Clang process, try building the second one instead of just sleeping?)

It might also be interesting to investigate whether we can reduce the 5 module variants produced by ninja clang down to just one, and whether there’s a worthwhile performance improvement from reducing the thousands of standard library PCM files to a smaller number.

[aengelke]

Thanks for looking into this. While your patch seems to reduce the number of errors, unfortunately I’m still unable to get a working build with -j96. The build keeps crawling and failing (with different kinds of module-related errors and some crashes) along with mostly just one process doing something; after 10 minutes I stopped it at 1102/2814.

With -j16, I did get a working build. Looks like there are still/other race conditions?

Build times of LLVM on the same machine with the same config as above with -j16 (note that CPU times are much lower than above as there’s no hyper-threading involved here):

• Regular: 355s (5586 CPU secs)
• PCH (PR linked above): 176s (2712 CPU secs)
• Modules cold: 244s (2516 CPU secs)
• Modules warm: crashes (module 'LLVM_IR' is defined in both)

The CPU time improvement of modules over 3 pre-compiled headers is less than 10%; the wall time is much higher due to reduced parallelism; and the modules build seems to be much less reliable.

[rnk]

I discussed the possibility of fixing up the implicit modules build in the Modules working group call this morning, but I wasn’t able to find a path or any volunteers interested in helping with that effort.

I decided to run some local experiments, and I found them very disappointing. I ended up with 15 directories in build/modules.cache, so core modules like LLVM_IR and LLVM_Support_DataTypes_Src get rebuilt 6-10 times throughout the build, causing random bottlenecks and low CPU utilization. With 16 cores, my implicit module build took 22min vs 558s for the equivalent non-modular build.

It feels like something should be possible here, but it’s going to be a long haul, so I withdraw my suggestion.