I’ve thought the opposite of having multiple tablegen tools, but have one tablegen invocation perform all of the generators at once and in parallel. There’s a lot of time spent in common building target information paths, only some of which gets used depending in the output type.
Also there’s probably low hanging fruit to speed up GlobalISelEmitter. I’ve never looked at the profile for it.
Also there’s probably low hanging fruit to speed up GlobalISelEmitter. I’ve never looked at the profile for it.
The problem here is that GlobalISelEmitter.cpp is slow to compile, not that llvm-tblgen -gen-global-isel is slow to run.
According to the graph GlobalISelEmitter.cpp is on the critical path. Having two or more llvm-foo-tblgen executables should add some parallelism.
An easy solution might be splitting GlobalISelEmitter.cpp
If we add more tablegen-* binaries, do we also need to increase the default number of parallel link jobs? Will adding more binaries make the build slower on systems with fewer cores?
That is another challenge with the LLVM build system. Do you tune for 1 or 96 cores?
Thank you everyone who gave me comments.
In 2021, I took an option to plugin-ize tblgen, since I wanted to avoid intrusive changes in the tree. Then I didn’t think I could add other tblgen executables.
As I read comments, I think it would be yet another option to add specific tblgen(s), if it would be acceptable.
Although decoupling GISelEmitter would be effective, I think introducing “CodeGen’s tblgen” would be better. I will work if I have my time.
@jrtc27 I have looked around llvm-tblgen and MVT.
I expected CodeGen stuff could be split out easily, but I knew also intrinsics_gen depends on MVT.
Could we use ValueTypes.td directly from other td(s) to avoid using MachineValueType.h, at least for IntrinsicsEmitter?
I guess it would not be easy to rewrite all emitters since we would have to rewrite many emitters.
For now, I suspend considering ValueTypes.td.
@tschuett @nhaehnle Let me know why you don’t prefer plugin.
(As I said before, it should be an option)
@tstellar tblgen executables would be expected smaller.
With a few libraries to link each tblgen executable.
If we would build plugins, each plugin module would be a few object files with a few dynamic libraries.
I think it would not be a problem unless we would try linking hundred of tblgen(s).
Because dlopen is not fun. Support for dlopen over all platforms supported by LLVM is even less fun (icky).
I’ve made a match proposing to move ItaniumManglingCanonicalizer/SymbolRemappingReader to ProfileData here: ⚙ D143318 [Support] Move ItaniumManglingCanonicalizer and SymbolRemappingReader from Support to ProfileData
I have introduced llvm-cg-tblgen internally, to isolate GISel and other codegen emitters. It certainly reduces the latency.
At first, I will isolate codegen stuff from IntrinsicEmitter.
https://reviews.llvm.org/D143844
(Oh excuse me, who is eligible to be a reviewer?)
to answer this specific question, I think anyone is eligible. Usually we look into the commit history (of the files you changed) and tag people who are most relevant as the reviewers. Though not always, you can also tag the code owner.
chapuni has been associated with LLVM for a long time, so I think he understands who’s eligible. But his native language is Japanese. I think the real question is, who would be an appropriate reviewer.
I wondered when I implemented llvm-cg-tblgen. I noticed it would make harder to configure llvm. (Imagine LLVM_TABLEGEN)
LLVM_CG_TABLEGEN in CMake cache. I think it is hairy.llvm-lite-tblgen as a subset of llvm-tblgen to satisfy building critical path (and hopefully MVT in the future) in-tree. Build and export llvm-tblgen as the fullset to build CodeGen.Opinions?
@pogo59 Thank you for correcting my improper wording. (To be honest, I checked up dictionaries then)
@mshockwave I think the most possible candidate would be “sabre” 
to me. I was really confused when I checked up the log.
Actually - we ran into this issue last year… Clang-tools-extra ended up with 2 new build-time executables, which needed similar variables for pointing out such a preexisting binary; we added similar cmake variables for pointing out those executables.
But keeping track of all new build-time executables is a bit of a mess, which is why we recently added a new, different cmake variable - LLVM_NATIVE_TOOL_DIR ([CMake] Allow setting the location of host tools with LLVM_NATIVE_TOO… · llvm/llvm-project@d3da906 · GitHub, and docs added in [docs] Add/update docs regarding LLVM_NATIVE_TOOL_DIR vs LLVM_TABLEGEN · llvm/llvm-project@95a2aec · GitHub) - which lets you set only one variable, and all build-time executables can be picked up from it.
With that setting available, it shouldn’t be too much of an issue with respect cmake settings, if we’d split the tblgen tools further.
I have created another patch.
https://reviews.llvm.org/D144351
It shall make things easier;
I have posted practical changes in TableGen.
This should be 1st step to generate MVT from ValueTypes.td.
Since I have committed D143844 as llvmorg-17-init-4919-g59fe64ae47d2, IntrinsicEmitter has become independent from other CodeGen emitters.
This is an experiment of llvm-cg-tblgen.
llvm-tblgen (w/o CodeGen emitters) finishes building in 6 seconds.
I know many tests in test/TableGen should be updated to use llvm-cg-tblgen. I will propose patches in this weekend.
This is the top of test-depends. The gap has been reduced so much, 20s to 9s. It is a big win on manycore machines. You can see that CodeGen emitters (including GISelEmitter) are independent from intrinsics_gen.
llvm-tblgen spends 8s to build, more than the graph above. I guess due to OS scheduler, I/O bottleneck, or processor’s power management.)Thank you. Happy hacking!
I am working for unifying MachineValueType.h and ValueTypes.td.
I have introduced llvm-min-tblgen. (⚙ D146352 Introduce `llvm-min-tblgen` to build public header files, and some fixups) This is dedicated for generating public header files.
Thanks to @arsenm, @dblaikie, @mstorsjo, and guys who gave me feedback.
The longest chain will be;
llvm-min-tblgenvt_genLLVMGlobalISelllvm-tblgen