Last week I committed a patch adding a new intrinsic for AMDGPU, and noticed a few unusual buildbot failures where we failed to select some other, unrelated intrinsics (this happened on an AMDGPU bot, but also on s390 and x86).
This was very difficult to reproduce - most of the time, things just pass. But I did manage to get one of these unlucky builds, and after digging around a bit I think the issue is that cmake doesn’t always regenerate AMDGPUGenDAGISel.inc and other target-specific TableGenerated files. It does however regenerate IntrinsicsAMDGPU.h, which means that the IntrinsicIDs become out of sync with the isel code. You can see one of these build logs here (but I’m also attaching it because it’s a pretty old build by now so it might disappear).
Has anyone else noticed anything similar? Since I didn’t manage to find a reliable way to run into this issue, I’m suspecting either a cmake bug (the buildbot is using cmake 3.22.1) or an issue with how we declare dependencies for TableGen targets. ninja -t deps doesn’t work on AMDGPUGenDAGISel.inc, but when I run ninja -t commands for it I do see the commands for generating IntrinsicsAMDGPU.h, so I expected AMDGPUGenDAGISel.inc to be regenerated whenever IntrinsicsAMDGPU.h is.
I don’t really understand the code in llvm/cmake/modules/TableGen.cmake, so I’m not sure if we’re correctly adding intrinsics_gen as a dependency for the individual TableGen commands. It would really mean a lot if someone more familiar with it could have a look.
I’m not familiar with these specific targets, but in the past (during my “Flang days”) we’d occasionally hit similar issues - failing on one bot but fine elsewhere, and almost impossible to reproduce. In nearly all cases, it turned out to be missing or incorrect CMake dependencies. Things “worked” most of the time purely out of luck.
So, as a first step, I’d try to track down and reason about the relevant dependencies. Hopefully, someone who has worked on these targets can offer a more specific suggestion.
-Andrzej
Seems like this bot is doing an incremental build? That is: not starting from a clean build directory and running cmake?
This is unsafe in general unfortunately, I’ve seen many issues that are hard-to-impossible to debug in this condition: bots should use ccache and not do incremental build to be safe (ccache is really fast!).
Ninja in particular is keep locally a .ninja_deps that is generated during the previous build and use it for tracking dependencies on rebuild, I never understood how ninja handles it’s invalidation because I wonder if this isn’t stalled somehow?
Now for your specific case, it is weird because llvm/include/llvm/IR/IntrinsicsAMDGPU.td was touched by the changed in the current commit and it should trigger a rebuild of AMDGPUGenDAGISel.inc but we don’t see it in the log.
And AMDGPUGenDAGISel.inc is a dependency of AMDGPUISelDAGToDAG.cpp.o which is rebuilt.
For me locally: ninja lib/Target/AMDGPU/AMDGPUGenDAGISel.inc -t deps will list a lot of files, including build/include/llvm/IR/IntrinsicsAMDGPU.h.
Ok, thanks for looking into it! I didn’t realize it was supposed to be ninja <target> -t deps instead of the other way around. I can see the dependence on IntrinsicsAMDGPU.h too now, so I guess there isn’t much we can do in CMake to fix this.