[FYI] CMake's Ninja generator is non-deterministic

It is with great sadness that I must tell everyone CMake’s Ninja generator is non-deterministic (https://public.kitware.com/Bug/view.php?id=15968).

I’m not sure if this impacts all versions of CMake, but it certainly impacts all the recent releases. You might ask why this matters? Sadly the non-determinism does impact determinism in the final builds. I haven’t fully dug into the extent of the differences, but I was seeing non-reproducibility in LTO builds. Whatever Ninja is doing differently causes LTO to make different optimization decisions. I have verified that this does not impact CMake’s makefile generator.

The important takeaway from this is that if you are building for a purpose where non-determinism can impact your results, don’t use Ninja. For most engineering tasks building and testing locally Ninja, as always, works great. For incremental bots testing correctness, no problem.

Any bots testing performance need to move to a deterministic generator (i.e. Unix Makefiles). Anyone building or testing a release of LLVM or clang for packaging purposes should also use a deterministic generator.

Sorry to be the bearer of bad news…

-Chris

From what I see reproducing offline, it is the order of dependencies that is changing (I bet they iterate over an unordered container):

Example:

< build cmake_order_depends_target_llvm-profdata: phony || lib/libLLVMSupport.a lib/libLLVMCore.a lib/libLLVMBitReader.a lib/libLLVMMC.a lib/libLLVMMCParser.a lib/libLLVMObject.a lib/libLLVMProfileData.a

Brad King has already replied with a patch that removes many of the differences (but not all). It was iterating over a std::map keyed on pointers.

-Chris

For context. I’ve tested the patch on the bug. It does resolve the non-determinism that matters. So if you have a locally built CMake, and you apply that patch, you can do a 3-stage build and compare with Ninja.

-Chris