Ninja can now check for missing CMake dependencies on generated files

LLVM Project
1

Have you ever noticed a flaky build with missing dependency errors like this?

D:\dev\projects\llvm-project\mlir\include\mlir/Dialect/Arith/IR/Arith.h(40): fatal error C1083: Cannot open include file: 'mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc': No such file or directory ninja: build stopped: subcommand failed.

As of Ninja version 1.11+, the ninja -t missingdeps tool can be used to find cases where generated source files (i.e. *.h.inc files produced by tablegen) were not added correctly to dependencies. Similar issues and tools for spotting them have been discussed on GitHub here: Method for detecting missing dependency graph edges · Issue #1660 · ninja-build/ninja · GitHub.

  • Certain missing generated file dependencies aren’t usually an issue since some other part of the build generates the files before they are needed. However, if you build on a smaller machine without much parallelism or you build leaf targets directly, these can cause flaky builds.

I ran Ninja’s missingdeps tool on LLVM and MLIR and I’m quite happy with the workflow and findings (and other LLVM subprojects could also find this useful):

  1. Configure

    D:\dev\projects\llvm-project (main -> upstream)
λ cmake -G Ninja -S llvm -B ./build-mlir/ \
    -DLLVM_ENABLE_PROJECTS=mlir \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo

  2. Build

    D:\dev\projects\llvm-project (main -> upstream)
λ ninja -C build-mlir all

  3. Check missing deps

    D:\dev\projects\llvm-project (main -> upstream)
λ ninja -C build-mlir -t missingdeps

Missing dep: tools/mlir/lib/Dialect/ArmNeon/CMakeFiles/obj.MLIRArmNeonDialect.dir/IR/ArmNeonDialect.cpp.obj uses tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Target/LLVMIR/CMakeFiles/obj.MLIRTargetLLVMIRExport.dir/LoopAnnotationTranslation.cpp.obj uses include/llvm/Frontend/OpenMP/OMP.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Target/LLVMIR/CMakeFiles/obj.MLIRTargetLLVMIRExport.dir/ModuleTranslation.cpp.obj uses include/llvm/Frontend/OpenMP/OMP.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Target/LLVMIR/CMakeFiles/obj.MLIRTargetLLVMIRExport.dir/Dialect/OpenMPCommon.cpp.obj uses include/llvm/Frontend/OpenMP/OMP.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Interfaces/CMakeFiles/obj.MLIRTilingInterface.dir/TilingInterface.cpp.obj uses tools/mlir/include/mlir/Dialect/Utils/DialectUtilsEnums.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Conversion/MemRefToLLVM/CMakeFiles/obj.MLIRMemRefToLLVM.dir/MemRefToLLVM.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Conversion/MemRefToLLVM/CMakeFiles/obj.MLIRMemRefToLLVM.dir/MemRefToLLVM.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Conversion/LinalgToStandard/CMakeFiles/obj.MLIRLinalgToStandard.dir/LinalgToStandard.cpp.obj uses tools/mlir/include/mlir/Dialect/Vector/Transforms/VectorTransformsEnums.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Conversion/NVVMToLLVM/CMakeFiles/obj.MLIRNVVMToLLVM.dir/NVVMToLLVM.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Conversion/NVVMToLLVM/CMakeFiles/obj.MLIRNVVMToLLVM.dir/NVVMToLLVM.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/OpenMP/CMakeFiles/obj.MLIROpenMPDialect.dir/IR/OpenMPDialect.cpp.obj uses include/llvm/Frontend/OpenMP/OMP.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/MLProgram/Transforms/CMakeFiles/obj.MLIRMLProgramTransforms.dir/PipelineGlobalOps.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/MLProgram/Transforms/CMakeFiles/obj.MLIRMLProgramTransforms.dir/PipelineGlobalOps.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.obj uses tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/OutlineShapeComputation.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/OutlineShapeComputation.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/RemoveShapeConstraints.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/RemoveShapeConstraints.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/ShapeToShapeLowering.cpp.obj uses tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/ShapeToShapeLowering.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOpsDialect.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/ShapeToShapeLowering.cpp.obj uses tools/mlir/include/mlir/Dialect/Func/IR/FuncOps.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/test/lib/Dialect/NVGPU/CMakeFiles/MLIRNVGPUTestPasses.dir/TestNVGPUTransforms.cpp.obj uses tools/mlir/include/mlir/Dialect/Linalg/Passes.h.inc (generated by CUSTOM_COMMAND)
Missing dep: tools/mlir/test/lib/Dialect/NVGPU/CMakeFiles/MLIRNVGPUTestPasses.dir/TestNVGPUTransforms.cpp.obj uses tools/mlir/include/mlir/Dialect/Vector/Transforms/VectorTransformsEnums.h.inc (generated by CUSTOM_COMMAND)
Processed 8259 nodes.
Error: There are 15 missing dependency paths.
15 targets had depfile dependencies on 7 distinct generated inputs (from 1 rules)  without a non-depfile dep path to the generator.
There might be build flakiness if any of the targets listed above are built alone, or not late enough, in a clean output directory.

Sure enough, if I try building any of those targets in isolation from a clean build, I hit errors like the one above:

D:\dev\projects\llvm-project (main -> upstream)
λ cmake -G Ninja -S llvm -B ./build-mlir-broken/ \
    -DLLVM_ENABLE_PROJECTS=mlir \
    -DCMAKE_BUILD_TYPE=RelWithDebInfo

D:\dev\projects\llvm-project (main -> upstream)
λ ninja -C build-mlir-broken MLIRArmNeonDialect

ninja: Entering directory `build-mlir-broken'
...
[882/883] Building CXX object tools\mlir\lib\Dia...MLIRArmNeonDialect.dir\IR\ArmNeonDialect.cpp.obj FAILED: tools/mlir/lib/Dialect/ArmNeon/CMakeFiles/obj.MLIRArmNeonDialect.dir/IR/ArmNeonDialect.cpp.obj
D:\dev\projects\llvm-project\mlir\include\mlir/Dialect/Arith/IR/Arith.h(40): fatal error C1083: Cannot open include file: 'mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc': No such file or directory
ninja: build stopped: subcommand failed.

In that case I can see that llvm-project\mlir\lib\Dialect\ArmNeon\IR\ArmNeonDialect.cpp has #include "mlir/Dialect/Vector/IR/VectorOps.h" but llvm-project\mlir\lib\Dialect\ArmNeon\CMakeLists.txt is missing a dependency on MLIRVectorDialect. Adding the missing dep and re-running the build (and missingdeps) confirms the fix worked.

Would anyone like to help fix these and perhaps include this check in a CI build so the changes stick?

10 Likes
2

This is just fantastic! :slight_smile:

I can promise to add this to the CI when the tree is clean.

I think we can go further and remove so big hammer fix we made at some point to force some extra automatic dependency in the graph because we couldn’t maintain these correctly.

3

I wonder if there is a similar tool but for link-time dependencies.

4

Building with -DBUILD_SHARED_LIBS=ON in cmake produces shared library files instead of static libraries that enforce all link dependencies being explicitly/properly specified. There are also build bots that test this configuration I believe.

5

It may trigger the issue (as LLVM_BUILD_LLVM_DYLIB may hide it), but the result won’t be really easy to take action on. I.e., it’s good to keep the build from slipping but doesn’t help untangle something with already missing dependencies. You’ll just get regular “missing symbol definition” or similar unhelpful linker errors.
Something that would say where the symbol actually is (which dependency to add) and help diagnose cyclic dependencies (A uses symbol from B which uses symbol from C which uses symbol from A, all works fine using default bfd linker but fails with lld since it’s stricter/uses different order - actual situation I’ve encountered) would be great.

6

Note that Ninja here does not provide anything better, it just indicates what’s missing (and it should have the info to provide the missing target I think?).

That said when I encountered “missing symbol definition” I find it pretty easy to figure out with a git grep <name> in the code name. Or alternatively for lib in lib/*.so ; do nm $lib | grep <name> && echo '$lib has <name>" ; done.

1 Like
7

One fix is here: [mlir][arith] Add ArithOpsInterfaces to mlir-generic-headers by anemet · Pull Request #70862 · llvm/llvm-project · GitHub

According to ninja -t missingdeps it fixes the three following issues:

< Missing dep: tools/mlir/lib/Dialect/ArmNeon/CMakeFiles/obj.MLIRArmNeonDialect.dir/IR/ArmNeonDialect.cpp.o uses /build/tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc (generated by CUSTOM_COMMAND)
< Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.o uses /build/tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc (generated by CUSTOM_COMMAND)
< Missing dep: tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/ShapeToShapeLowering.cpp.o uses /build/tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc (generated by CUSTOM_COMMAND)
(jax_venv)

However even before the fix I can’t reproduce the issue:

$ rm tools/mlir/include/mlir/Dialect/Arith/IR/ArithOpsInterfaces.h.inc
$ ninja tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.o  
[2/2] Building CXX object tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.o

It correctly regenerate the ArithOpsInterfaces.h.inc before building one of the faulty target.

@scotttodd : any hint?

(@anemet for vis)

8

I wasn’t able to reproduce that either :thinking:

$ git checkout 2164a449dc7a9 # (prior to the commit from that PR landing)
$ rm -rf ./build-mlir
$ cmake -G Ninja -S llvm -B ./build-mlir/ -DLLVM_ENABLE_PROJECTS=mlir -DCMAKE_BUILD_TYPE=RelWithDebInfo
$ ninja -C build-mlir MLIRShapeOpsTransforms

...
[1596/1596] Linking CXX static library lib\MLIRShapeOpsTransforms.lib
# (success)

FWIW, I also ran missingdeps on IREE downstream (logs here) and saw a subset of the LLVM/MLIR warnings along with a few downstream-specific warnings. This mechanism for detecting missing dependencies is pretty reliant on what targets were already built, unlike the more rigorous dependency modeling in build systems like Bazel.

9

I think it may be that it is only considering via deps and so if there is a link dep then its not considered as a dep while the link dep ensures the build order. (at least the change that makes it happy for Shape here that I’ll submit seems to point this way).

10

A link dep wouldn’t trigger during the build AFAIK?

That is when I run ninja tools/mlir/lib/Dialect/Shape/Transforms/CMakeFiles/obj.MLIRShapeOpsTransforms.dir/Bufferize.cpp.o the link-only dependency shouldn’t be built.

Also ninja has visibility: it should understand the graph and “should know” that when I call this target above, the generated file or will not be produced.