Issues in llvm-tblgen -- High-parallelized build

[tschuett]

@chapuni

@tschuett @nhaehnle Let me know why you don’t prefer plugin.
(As I said before, it should be an option)

Because dlopen is not fun. Support for dlopen over all platforms supported by LLVM is even less fun (icky).

[RKSimon]

I’ve made a match proposing to move ItaniumManglingCanonicalizer/SymbolRemappingReader to ProfileData here: ⚙ D143318 [Support] Move ItaniumManglingCanonicalizer and SymbolRemappingReader from Support to ProfileData

[chapuni]

I have introduced llvm-cg-tblgen internally, to isolate GISel and other codegen emitters. It certainly reduces the latency.

At first, I will isolate codegen stuff from IntrinsicEmitter.
https://reviews.llvm.org/D143844
(Oh excuse me, who is eligible to be a reviewer?)

[mshockwave]

@chapuni

Oh excuse me, who is eligible to be a reviewer?

to answer this specific question, I think anyone is eligible. Usually we look into the commit history (of the files you changed) and tag people who are most relevant as the reviewers. Though not always, you can also tag the code owner.

[pogo59]

chapuni has been associated with LLVM for a long time, so I think he understands who’s eligible. But his native language is Japanese. I think the real question is, who would be an appropriate reviewer.

[chapuni]

I wondered when I implemented llvm-cg-tblgen. I noticed it would make harder to configure llvm. (Imagine LLVM_TABLEGEN)

• Introduce LLVM_CG_TABLEGEN in CMake cache. I think it is hairy.
• Build llvm-lite-tblgen as a subset of llvm-tblgen to satisfy building critical path (and hopefully MVT in the future) in-tree. Build and export llvm-tblgen as the fullset to build CodeGen.

Opinions?

@pogo59 Thank you for correcting my improper wording. (To be honest, I checked up dictionaries then)

@mshockwave I think the most possible candidate would be “sabre” to me. I was really confused when I checked up the log.

[mstorsjo]

@chapuni

I wondered when I implemented llvm-cg-tblgen. I noticed it would make harder to configure llvm. (Imagine LLVM_TABLEGEN)

• Introduce LLVM_CG_TABLEGEN in CMake cache. I think it is hairy.

Actually - we ran into this issue last year… Clang-tools-extra ended up with 2 new build-time executables, which needed similar variables for pointing out such a preexisting binary; we added similar cmake variables for pointing out those executables.

But keeping track of all new build-time executables is a bit of a mess, which is why we recently added a new, different cmake variable - LLVM_NATIVE_TOOL_DIR ([CMake] Allow setting the location of host tools with LLVM_NATIVE_TOO… · llvm/llvm-project@d3da906 · GitHub, and docs added in [docs] Add/update docs regarding LLVM_NATIVE_TOOL_DIR vs LLVM_TABLEGEN · llvm/llvm-project@95a2aec · GitHub) - which lets you set only one variable, and all build-time executables can be picked up from it.

With that setting available, it shouldn’t be too much of an issue with respect cmake settings, if we’d split the tblgen tools further.

[chapuni]

I have created another patch.
https://reviews.llvm.org/D144351

It shall make things easier;

• To move files to other tablegen executables.
• To introduce emitter plugins.

[chapuni]

I have posted practical changes in TableGen.

[RFC] Let Intrinsics.td emit IIT_Info TypeSig

I have noticed that logics of generating TypeSig in IntrinsicEmitter can be migrated to Intrinsics.td, with a few enhancements in TableGen, while working on TableGen. My motivation is to let IntrinsicEmitter free from MachineValueType.h and migrate other CodeGen emitters to another TableGen like llvm-cg-tblgen. I have noticed that MVT is mostly used in the logic of TypeSig in IntrinsicEmitter. I would like to propose changes along steps below. Preparations Split out CodeGenIntrinsics from…

This should be 1st step to generate MVT from ValueTypes.td.

[chapuni]

Since I have committed D143844 as llvmorg-17-init-4919-g59fe64ae47d2, IntrinsicEmitter has become independent from other CodeGen emitters.

This is an experiment of llvm-cg-tblgen.
llvm-tblgen (w/o CodeGen emitters) finishes building in 6 seconds.

I know many tests in test/TableGen should be updated to use llvm-cg-tblgen. I will propose patches in this weekend.

Commits · chapuni/llvm-project

This is the top of test-depends. The gap has been reduced so much, 20s to 9s. It is a big win on manycore machines. You can see that CodeGen emitters (including GISelEmitter) are independent from intrinsics_gen.

(llvm-tblgen spends 8s to build, more than the graph above. I guess due to OS scheduler, I/O bottleneck, or processor’s power management.)

Thank you. Happy hacking!

[chapuni]

I have created the diffusion for llvm-cg-tblgen.
https://reviews.llvm.org/D146352

[chapuni]

I am working for unifying MachineValueType.h and ValueTypes.td.

[RFC] Let MVT generated and restore MVT into llvm/CodeGen

I have been working for TableGen and MachineValueType.h. I have confirmed that MVT can be generated from ValueTypes.td and MVT and LLT in llvm/Support can be restored to llvm/CodeGen. Generate MVT from ValueTypes.td Implement VTEmitter for MVT The series of D146179 ValueTypes.td: Reorganize ValueType to generate `MachineValueType.h` Let IntrinsicEmitter independent from MVT The series of D145937 Replace `IntrinsicEmitter::ComputeFixedEncoding()` and cleanup Split llvm-tblgen into …

[chapuni]

I have introduced llvm-min-tblgen. (⚙ D146352 Introduce `llvm-min-tblgen` to build public header files, and some fixups) This is dedicated for generating public header files.

Thanks to @arsenm, @dblaikie, @mstorsjo, and guys who gave me feedback.

The longest chain will be;

1. llvm-min-tblgen
2. vt_gen
3. LLVMGlobalISel
4. llvm-tblgen
5. (the longest target module)