Organization of LLVM utilities

I was confused by the lack of TableGen backends in the llvm/lib/tablegen directory until I stumbled upon the llvm/utils/tablegen directory. Could someone explain why TableGen is divided into these two directories?

I am assuming the TableGen code is split like this so that external tools can utilize the TableGen parser without all of the specific logic in the LLVM tablegen binary. I can't find any application that uses it though aside from tablegen proper.

Lib is for the core implementation of the language. The different features in utils are the different backends that all interpret it in different ways.

-Matt

There is also clang/utils/TableGen which contains the clang tablegen backends.The build produces an llvm-tblgen from llvm/utils/TableGen and clang-tblgen from clang/utils/TableGen

The mechanical reason is that llvm/utils builds before llvm/libs. The basic TableGen support is required to generate .inc files in the libs directory.

-Chris

Could you expand on this a bit? I see from my build directory that llvm-tblgen.exe was built before the Target/* .inc files. It sounds like you're saying that something has to be built between the .exe and the .inc files.

I may add that MLIR is doing the same thing as clang, reusing the TableGen language without the LLVM specific backends.
We have quite a few generator, from the mlir-tblgen --help output:

Generator to run
–gen-attr-interface-decls - Generate interface declarations
–gen-attr-interface-defs - Generate interface definitions
–gen-attr-interface-docs - Generate interface documentation
–gen-avail-interface-decls - Generate availability interface declarations
–gen-avail-interface-defs - Generate op interface definitions
–gen-dialect-decls - Generate dialect declarations
–gen-dialect-doc - Generate dialect documentation
–gen-enum-decls - Generate enum utility declarations
–gen-enum-defs - Generate enum utility definitions
–gen-enum-from-llvmir-conversions - Generate conversions of EnumAttrs from LLVM IR
–gen-enum-to-llvmir-conversions - Generate conversions of EnumAttrs to LLVM IR
–gen-llvmir-conversions - Generate LLVM IR conversions
–gen-llvmir-intrinsics - Generate LLVM IR intrinsics
–gen-op-decls - Generate op declarations
–gen-op-defs - Generate op definitions
–gen-op-doc - Generate dialect documentation
–gen-op-interface-decls - Generate interface declarations
–gen-op-interface-defs - Generate interface definitions
–gen-op-interface-docs - Generate interface documentation
–gen-pass-decls - Generate operation documentation
–gen-pass-doc - Generate pass documentation
–gen-rewriters - Generate pattern rewriters
–gen-spirv-avail-impls - Generate SPIR-V operation utility definitions
–gen-spirv-capability-implication - Generate utility function to return implied capabilities for a given capability
–gen-spirv-enum-avail-decls - Generate SPIR-V enum availability declarations
–gen-spirv-enum-avail-defs - Generate SPIR-V enum availability definitions
–gen-spirv-op-utils - Generate SPIR-V operation utility definitions
–gen-spirv-serialization - Generate SPIR-V (de)serialization utilities and functions
–gen-struct-attr-decls - Generate struct utility declarations
–gen-struct-attr-defs - Generate struct utility definitions
–gen-type-interface-decls - Generate interface declarations
–gen-type-interface-defs - Generate interface definitions
–gen-type-interface-docs - Generate interface documentation

It may be confusing but it is not the case that “llvm/lib” contains all the libraries, and “llvm/tools” contains all the tools. At a course grain, the main llvm repo builds in four phases:

1) llvm/utils. -> this builds some executables like tablegen
2) llvm/lib. -> This is all libraries, shouldn’t include executables.
3) llvm/tools. -> Generally executables, also some libraries that are tool specific.
4) tests.. -> Things that depend on the above.

In theory, we could have a much finer grain build system that allows tracking exactly which tests depend on which executables, which executables depend on which libraries, etc, but this hasn’t been done. In theory this could help the build cycle if you’re running a single “llvm-mc” like test, for example.

-Chris

It may be confusing but it is not the case that “llvm/lib” contains all the libraries, and “llvm/tools” contains all the tools. At a course grain, the main llvm repo builds in four phases:

1) llvm/utils. -> this builds some executables like tablegen
2) llvm/lib. -> This is all libraries, shouldn’t include executables.
3) llvm/tools. -> Generally executables, also some libraries that are tool specific.
4) tests.. -> Things that depend on the above.

In theory, we could have a much finer grain build system that allows tracking exactly which tests depend on which executables, which executables depend on which libraries, etc, but this hasn’t been done. In theory this could help the build cycle if you’re running a single “llvm-mc” like test, for example.

-Chris

llvm/test/CMakeLists.txt LLVM_TEST_DEPENDS lists many utilities. If I
understand correctly, every test under llvm/test/ requires to build all
tools listed in LLVM_TEST_DEPENDS. This is probably fine because these
utilities are all small.

Until people pointed it out, it hadn't registered that of course TableGen is not just one table generator, but many, depending on the backends attached. So the common code is separate from the various backends for at least that reason.