[RFC] Declare library command line options in TableGen, one struct per library

[MaskRay]

cl::opt globals are how LLVM’s libraries declare their internal developer options (2800+ in 70 components of llvm/lib; 1749 are cl::opt<bool>).
Each one is a constructor run at startup that registers into a StringMap, ~200 bytes of .bss and a couple of load-time relocations; the value is process-wide, so two LLVMContexts in one process cannot differ (rustc’s -Cllvm-args is silently dropped after the first session for this reason); and a library reading another’s option writes extern cl::opt<bool> X; (147 in tree, 48 across libraries) or exports one from a public header.

RFC: replacement CLI library for LLVM and tools proposes a new library, CommandLineV2, to fix this.
This proposal fixes it with what we already have: llvm/Option’s TableGen representation (recently shrunk from 88 bytes to 16 bytes and zero dynamic relocations per option, #224374 through #227582, thanks to @aengelke for all the reviews) for the declarations, and a generated struct per library for the values. A context (LLVMContext for the IR-level libraries, MCContext for the MC layer) holding per-context copies is a later step.

cl:: keeps owning command line parsing until the migration completes, so option spellings do not change for tools.

Status: the cl:: cleanup (step 1 below) has landed, and so has the framework (#226087, with LLVMCGData’s 13 options). InstCombine (#227948) and Sparc (#227977) followed, chosen to stress the framework: a pass library with a central object, and a target whose options reach code through the TargetMachine and Subtarget. X86 and MC are next; they add std::optional, enum and tri-state fields.

Declaring

A library’s options live in a .td in its own tree:

include "llvm/Option/LibraryOptions.td"

def CGDataOptions : OptionsStruct;

defm : BoolField<"codegen-data-generate", "false",
  "Emit CodeGen Data into custom sections">;
defm : ValueField<"codegen-data-use-path", "StringRef", "",
  "File path to where .cgdata file is read">;
defm : ValueField<"global-merging-max-params", "unsigned",
  "std::numeric_limits<unsigned>::max()",
  "The maximum number of parameters allowed when merging functions.">;

llvm-tblgen -gen-opt-parser-defs emits struct CGDataOptions { bool codegen_data_generate{0}; StringRef codegen_data_use_path{}; ... }, a Global instance, the option table, and apply(const Arg &).
A member is named after its option in snake_case, matching OptTable’s OPT_codegen_data_generate; defm Name : ... names it explicitly. A default is the member’s C++ initializer.
OptionsStruct<"mlir"> places the struct in another namespace, and OptionsStruct<prefix = "sparc-"> drops a shared prefix from member names, so -sparc-bpcc-offset-bits sets bpcc_offset_bits while the spelling keeps its prefix.

The rest of a home is fixed boilerplate, the same for every library; this is all of LLVMCGData’s, license banners removed:

// lib/CGData/CGDataOptions.h
#include <limits>
#define OPTIONS_STRUCT_DECL
#include "CGDataOptions.inc"

// lib/CGData/CGDataOptions.cpp
#include "CGDataOptions.h"
#include "llvm/Option/LibraryOptions.h"
#define OPTIONS_STRUCT_DEFS
#include "CGDataOptions.inc"

static llvm::opt::RegisterLibraryOptions<llvm::CGDataOptions> Registration;

A target adds no files besides its .td: XXXSubtarget.h includes the declaration block, XXXSubtarget.cpp the definitions, and LLVMInitializeXXXTarget() registers with a function-local static opt::RegisterLibraryOptions<XXXOptions> O;, so a target’s options cost no static initializer and exist only once the target is initialized. llc, opt, clang -cc1 and lld initialize targets before parsing -mllvm (lld/ELF needed a one-line reorder).

# lib/CGData/CMakeLists.txt
set(LLVM_TARGET_DEFINITIONS CGDataOptions.td)
tablegen(LLVM CGDataOptions.inc -gen-opt-parser-defs)
add_public_tablegen_target(CGDataOptionsTableGen)
# and DEPENDS CGDataOptionsTableGen, LINK_COMPONENTS Option on the library

Adding an option is one defm.
A bool is -x or -x=true|false|1|0, and a value is -x=v or -x v, exactly what cl::opt accepts, so no test or script changes spelling when a library migrates. --x works for -x.
Library options are developer knobs, so only -help-hidden lists them (2,589 of llvm/lib’s 2,939 cl::opts are already cl::Hidden). Options users should see belong in a tool’s OptTable, including the shared codegen flags such as -mtriple (CodeGen/CommandFlags.cpp), which become a .td fragment that tools include rather than a library struct.
Members may be bool, integers, float, double, or StringRef. StringRef rather than std::string keeps Global constant-initialized: no constructor, no destructor, no allocation for a non-empty default. Each further kind arrives with the first library that needs it:

• std::optional<T> defaulting to std::nullopt replaces getNumOccurrences() (X86).
• OptionalBoolField, a std::optional<bool>, replaces cl::boolOrDefault; EnumField replaces cl::values and shows its values as the metavar, -dwarf-extended-loc=<Default|Enable|Disable> (MC).
• DefaultOnOffField is an EnumField of std::optional<bool> spelled Default|Enable|Disable; it replaces the file-local DefaultOnOff enums in MC and DwarfDebug, so no library has to export one.
• Two new OptTable kinds, FlagOrEq (-x, -x=v) and SeparateOrEq (-x=v, -x v), make every field one table row instead of a Joined twin plus a Flag or a Separate alias; CGDataOptions.cpp.o shrinks from 4,977 to 4,370 bytes. lld/ELF’s Eq/EEq options have the same shape and drop 68 of 435 rows with them.

Lists come with the first library that needs one.

Reading

• CGDataOptions::Global.codegen_data_generate: one load, the same as a cl::opt read. This is the mechanical replacement for a static cl::opt, and what is left when a library has no owning object.
• Where an owner exists, it binds Global once and readers go through it. SparcTargetMachine holds const SparcOptions &CLOpts, SparcSubtarget copies the reference, and code reads Subtarget.getCLOpts().bpcc_offset_bits; InstCombine’s InstCombinerImpl does the same. Only the owner’s constructor names Global, so grep '::Global' is the list of sites a context has to reach.

A home lives in lib/X/ and is private, like the static cl::opts it replaces: another library that needs a knob calls a getter or takes a parameter (LTO calls cgdata::thinLTOTwoRounds(); llvm-cgdata passes lazy loading to CodeGenDataReader::create, #226861). A library can expose its struct under include/ when there is a real user, such as an embedder setting a value without building argv strings.

Toward per-context options

RegisterLibraryOptions does two jobs that separate cleanly. Registering the option table is immutable: it lets cl:: recognize, validate and list -mllvm -sparc-..., and it can stay global. Storing the values in T::Global is the global state. apply(const Arg &) is an instance method, so any owner can hold its own copy.

The path, in order:

1. Readers go through an owner. Done in each port; what still names Global is the inventory.
2. opt::parseLibraryOptions<T>(Args, Base) builds an instance from a base and an argument list, using T’s table and ignoring other libraries’ arguments.
3. The per-compilation arguments that tools already keep (clang’s FrontendOpts.LLVMArgs, lto::Config::MllvmArgs) reach the owner through its options: TargetOptions for a TargetMachine, MCTargetOptions for MC, LLVMContext or the pass builder for IR passes. The owner’s instance is Global with those arguments applied on top, so -mllvm on the process command line keeps working. For Sparc the switch is one line: CLOpts(getLibraryOptions<SparcOptions>()) in place of CLOpts(SparcOptions::Global).
4. A tool stops applying a library’s options globally once it hands them to the context; validation still goes through the registry, so unknown options are rejected once, up front.
5. Global goes when nothing reads it, and RegisterLibraryOptions shrinks to table registration.

Steps 1–3 can land today as a handful of small patches, but they change nothing visible: while tools still parse -mllvm globally, every context’s copy equals Global. What blocks step 4 is outside the ports: the -mllvm list also carries ~2,800 unported cl::opts, so tools keep calling cl::ParseCommandLineOptions; embedders (LLVMParseCommandLineOptions, rustc, Swift, Julia) set options process-wide and need Global as a fallback; and a library needs an owner that reaches its readers (targets and MC have one, IR passes need plumbing, CGData has none). So the plumbing waits for its first user — clangd or libclang reusing -cc1 in process, threaded ThinLTO backends with different settings, a JIT — and the ports keep the rule that makes it a one-line switch.

cl:: integration

RegisterLibraryOptions<T> hands cl:: the option names, a parser, help and reset.
ParseCommandLineOptions indexes the names at its first parse and dispatches an argument that names none of cl::'s options to the library that declares it, so --, subcommands and positional arguments keep cl::'s rules. The index is rebuilt lazily when a library registers during parsing. A name shared with a cl::opt or another library is a fatal error; options of unregistered subcommands (MLIR’s PassOptions) are not affected.
cl:: copies each library argument into an allocator that ResetAllOptionOccurrences resets, so StringRef members outlive response files. -help-hidden (under “Library options:”) and ResetAllOptionOccurrences cover library options; -print-options, “Did you mean” suggestions and cl::OptionCategory do not yet, so cl::HideUnrelatedOptions cannot hide them.

Cost

Measured on the prototype, which also migrated LLVMPasses: PassesOptions.cpp.o for 33 options was 13.6 KB of text (a 98-row, 24-byte table including the since-dropped -no-x rows, 3.9 KB of strings, the apply switch) and a 195-byte static initializer, against −6.6 KB of static-initialization code, −7.2 KB of .bss and −52 relocations for the 33 cl::opts it replaced. Reads compile to loads; the tracker run shows CTMark within noise (geomean ±0.01% instructions, ±0.26% max-rss) and the clang binary at −1.5% .bss, +0.01% text.

Migrating a library

Net line count is the honest metric for a migration. On the prototype, next to the CommandLineV2 branch’s migration of the same libraries:

	options	prototype	CommandLineV2 branch
LLVMPasses	33 (50 there, incl. StandardInstrumentations)	8 files, +246 −229 = +17	18 files, +1563 −496 = +1067
of which PassBuilderPipelines.cpp		+68 −224 = −156	+263 −233 = +30
LLVMCGData, LTO’s read, llvm-cgdata	13 (12 there)	11 files, +149 −101 = +48	16 files, +481 −123 = +358
per option		+0.5 (Passes), +3.7 (CGData)	+21, +30

Where the lines go here: the .td (the old cl::opt declarations with their help text, 2–3 lines each), a small header and .cpp that are mostly license, and a few build lines; the read sites shrink (static cl::opt<bool> X(...) + X becomes Opts.x). On the CommandLineV2 branch each read site gains a static bool getX(const OptionsContext &) wrapper and its callers gain a context parameter.

Extrapolated to the 2,806 options in llvm/lib at +0.5 to +4 net lines each, the whole migration is a few thousand lines, done one library at a time by its owners; the CommandLineV2 branch is +72k net for 3,547 options.

Relation to CommandLineV2

Both keep per-library .td files and aim for per-context values. The differences:

• No new parser or library: OptTable already exists, is used by clang, lld and most tools, and its per-option cost was just cut from 88 to 16 bytes.
• A read is a field load; CommandLineV2 reads through an open-addressed hash on the registry address (OptionsContext::findData) plus an assert-build check, which the tracker measures at +0.34% instructions stage1-O3, +0.90% ReleaseThinLTO, +0.80% ReleaseLTO-g.
• Per-context is incremental, not mandatory. CommandLineV2’s defaultOptionsContext() is empty, so an unthreaded read gets compile-time defaults and ignores the command line; the branch has 93 such reads and a mandatory LLVMContext(const OptionsContext &). Here Global is the parsed instance.
• Per-option cost in the .td: 2–3 lines against 11; a home is 3 files against 7; a read is Opts.x against a hand-written getter per option.

Plan

1. Remove the cl:: parser features no library option uses, so the two parsers agree. Landed: cl::Sink (#224953), cl::multi_val (#224959), cl::opt<bool> accepting only true|false|1|0 (#224975), cl::DefaultOption (#225609), cl::Grouping (#224958, after llvm-cov gcov moved to OptTable in #224955), LongOptionsUseDoubleDash (#226347, after yaml2obj #224964, obj2yaml #224965 and split-file #224966 moved to OptTable), cl::bits (#226628), and BOLT’s cl::callbacks (#226359). -no-<name> for bools (#224978) was withdrawn.
2. Land the framework with LLVMCGData as the first library (#226087).
3. Libraries migrate on their own schedule, one .td each; the recipe is in CommandLine.md. Landed: CGData, InstCombine, Sparc. Next: the one-row option kinds, X86, MC, then LLVMPasses. Option renames (dropping a redundant prefix) are separate, later patches, since they break scripts. Cross-library writers (DirectX’s DXILWriterPass assigning MC’s -dx-*) stay cl::opt until the writer takes a parameter. The remaining cl::callback users move first; pass plugins get their arguments through -plugin-arg=<plugin>,<arg> and are loaded after parsing (#226773).
4. Per-context copies, as above, starting where there is a user (rustc’s per-session -Cllvm-args needs LLVMPasses).
5. Tools keep moving to OptTable one at a time, as they already do. When no cl::opt is left, cl::ParseCommandLineOptions is a loop over the registered structs and cl:: can go.

MLIR

MLIR has no file-scope cl::opt in its libraries; its uses fall into five patterns:

• Option groups registered on demand (MLIRContext, AsmPrinter, PassManager, Timing, DebugCounter): each becomes an OptionsStruct registered from a function-local static in registerXCLOptions(), so MLIR’s -Werror=global-constructors holds. The ManagedStatic and every isConstructed() check go. A ported MLIRContextOptions reads --mlir-disable-threading with one compare instead of three loads, two null tests and a compare, and the MLIRContext constructor goes from 355 to 298 instructions. These need enum, list and std::optional fields.
• Options declared inside translation registrations (Cpp, LLVMIR, SPIRV, SMTLIB) port the same way; all are bool or string.
• Tool mains (MlirOptMain and the other *Main functions) belong to the tool phase; downstream tools call them and add their own cl::opts. --split-input-file’s implied marker is expressible as a Flag with a FieldValue.
• Registry-driven options (each registered pass as --<pass>, and translations likewise) stay on cl:: for now; cl::LibraryOptions is a runtime interface that the pass registry could implement directly.
• Pass options, parsed per instance from pipeline strings with cl::opt objects in private subcommands, are out of scope. They are already declared in TableGen (Passes.td), so mlir-tblgen could generate plain members and a parser for them, decoupling pipeline syntax from cl::.

Q&A

Multiple instances per process? Not yet: Global is process-wide, as cl::opt is. Readers already go through an owner, so the per-context step (above) switches one constructor per library. The known users (rustc’s per-session -Cllvm-args, JIT embedders) need this in LLVMPasses, not everywhere.

Tool-level flag registries? CodeGen/CommandFlags.cpp and MC/MCTargetOptionsCommandFlags.cpp are opt-in registrations that tools copy into TargetOptions/MCTargetOptions; they are already per-instance in shape and are not library structs. Their declarations can move to a .td that tools include, with std::optional for the getExplicit* accessors; that is a separate design.

Which contexts? Of llvm/lib’s 2,806 options, 2,644 are read with an LLVMContext at hand (IR, Analysis, Transforms, Passes, LTO, CodeGen, the targets’ codegen halves); ~100 are the MC layer (lib/MC, the targets’ MCTargetDesc/AsmParser/Disassembler) and belong to MCContext, which already holds MCTargetOptions; ~65 (Support, ProfileData, TableGen, CGData) have no context and read Global. Polly’s 156 are LLVMContext; BOLT’s 362 would be BinaryContext.

Plugins? A plugin’s struct registers when the DSO loads, as its cl::opts do, and cl:: reindexes a library registered during parsing. #226773 changes opt and llc to load plugins after parsing and pass them arguments with -plugin-arg=<plugin>,<arg>, so a plugin’s options no longer share the process’s namespace.

Support’s own cl::opts? Every tblgen links Support, so Support and lib/TableGen cannot use generated code. Their 25 options (-help, -debug-only, -stats, -debug-counter, tblgen’s -o/-I/-d, …) are tool-level and stay cl::opt for now. When they move, the existing bootstrap covers it: llvm-min-tblgen already generates for LLVMTableGenCommon, which llvm-tblgen links; so the OptParser emitter moves into TableGen/Basic, Support’s state-setting options (-debug-only, -stats, -debug-counter, the timer knobs) are declared in a library above Support that llvm-tblgen links, and llvm-min-tblgen’s own eight driver options get a minimal parser.

Why keep cl:: for now? Because it owns argv in every tool and in -mllvm, keeping it is what makes the migration incremental: a library moves when its owners have time, every tool works throughout, and nothing is flag-dayed. cl:: shrinks as libraries and tools leave it; the end state has no cl::opt.

---

Disclosure: the prototype, the measurements and this text were produced with Claude Code; the design decisions are mine. @aengelke’s comments and previous RFCs including [RFC] Replacement CLI library for LLVM and tools are in the context.

[steffenlarsen]

Thank you, @MaskRay. I agree that this looks like a solid alternative, avoiding too much redundant functionality. I would be happy to help make this happen. That said, based on the prototype, I have the following concerns/questions:

1. This solution seems to solve the global state issue, like the primary focus of the solution I proposed in the other RFC, but based on my reading it seems like any tool that uses a dynamically loaded library will also get all the options registered, while the other proposal let tools opt-in to plugin options (though it is all-or-nothing.) Am I missing something or do we deem this low-priority?
2. In the other RFC I intentionally avoided changes the behavior of existing options to reduce the intrusive surface of the changes. In contrast, this prototype changes how some of the options it touches are used (e.g. -enable-loopinterchange=true → -enable-loopinterchange). Is this a side-effect of the new interfaces or just a “while we are at it”-change? In either case, is this somethign we want to do, risking that some external tool breaks?
3. Migration of well-behaving options is one thing, but is this design able to handle options that use more colourful functionality, such as callbacks and runtime-adjusting option information?
4. The other RFC also uses TableGen, though it is not a requirement. One issue I ran into was trying to use TableGen with Support, as it caused a chicken-and-egg situation where TableGen needs Support to be built and Support needs TableGen to have generated its options files. Do you have an idea for how to solve this?
5. Maybe I missed this in the description, but is the intention to keep CommandLine, thin it after migration, or completely drop it eventually?

[MaskRay]

1. When a plugin DSO loads, its dynamic initialization (RegisterLibraryOptions<T>) will run and cause the subsequent ParseCommandLineOptions to parse its registered options among the other registered options. If this out to matter, the DSO can move to an explicit RegisterLibraryoptions<T> call.
2. This is an orthogonal “while we are at it” decision. While we can keep parsing -foo=false (while every other CLI library uses something like -no-foo), I suggest that we eventually reject this form. I’ve created [Support] Accept -no-<name> to negate cl::opt<bool> by MaskRay · Pull Request #224978 · llvm/llvm-project · GitHub to support -no-foo in the current, legacy cl:: . If this is accepted, I can request a release/23.x backport (a few rust-lang/rust tests use something like -Cllvm-args=-enable-global-merge=0). Anyway, internal options do not provide stability guarantee. This is for courtesy.
3. We should remove the cl::callback feature. They will become plain field assignment and surround code should inspect the state and make transformation.
4. They can stay cl::opt until the flag day, or we can move them from LLVMSupport to a different library. We can also hand write OptTable.
5. Per “Why keep cl:: for now?” at the end of Q&A: cl:: shrinks as libraries and tools leave it; the end state has no cl::opt

[steffenlarsen]

I’ll play around with it a bit, but I’m preliminarily fine if concensus is to follow this direction.

[aengelke]

Thanks for looking into this, I like this direction a lot more. I haven’t yet looked into all details, but the overall direction looks good to me.

Some data on compile times, max-rss, size-data, and size-bss changed would be nice. (I would hope that the extra indirections when accessing command line options, but one can never be sure…)

@MaskRay

• PassesOptions::Current.RunNewGVN

It would be good if “Current” would be a different name that is more easily and unambiguously greppable, as such uses indicate global option access that we want to phase out.

@MaskRay

A plugin’s struct registers when the DSO loads, as its cl::opts do. opt -load-pass-plugin=X.so -x-knob works: the parser indexes a library registered by an earlier argument. clang loads plugins before parsing -mllvm.

We could alternatively also stop adding plugin options to the global option namespace… to me this always seemd to be a weird design that was imposed by llvm::cl only supporting such use. While I added this code to support options in plugins as the easiest way to do so, we could also introduce some -plugin-arg or (probably more preferable?) go for some syntax like -load-pass-plugin=foo.so,-opt,-no-foo. I wouldn’t worry too much about supporting option registration while parsing options.

@MaskRay

if cl::opt is ever retired they become a hand-written OptTable, which the Info rows allow.

Not sure I like this bit, this would make it much harder to change the representation. Support options could also be moved elsewhere (I don’t think TableGen depends on Support parts that have options). For TableGen we could also hand-roll a very minimal option parser.

@steffenlarsen

3. Migration of well-behaving options is one thing, but is this design able to handle options that use more colourful functionality, such as callbacks and runtime-adjusting option information?

I agree with @MaskRay that we should remove such functionality.

@steffenlarsen

5. Maybe I missed this in the description, but is the intention to keep CommandLine, thin it after migration, or completely drop it eventually?

Given it’s uses in clang-tools-extra, polly, Flang, MLIR, Clang, Bolt, it’ll be a long time until we can consider to actually remove llvm::cl.

---

PS: CommandLineV2 is somewhat of a misnomer, llvm::cl is already version 2, introduced in 2002.

[jh7370]

I’m more of a fan of this one versus the other proposal, mostly because it sounds like we don’t ever have an interim period of having 3 different libraries for the same thing and it opens up a pathway to get it down to 1 gradually, which is a bonus (but not a requirement from my point of view). I’m not really that fussed by the details, so I’ll let others who are more familiar with that area drive that discussion.

[jansvoboda11]

I also prefer this to the original proposal. I think unifying how we work with command-line options across the project is a step forward.

One thing I’d like to highlight is that Clang currently declares the option struct in a separate header file where some of its members are explicitly initialized to a default value. The marshalling infrastructure then declares a separate default value in the TableGen file. I like your idea of generating the struct from the TableGen file directly, removing the possibility for conflicting default values. If this ends up being implemented in the library, I would like for Clang to adopt it as well.

[MaskRay]

Tracker run on a PR migrating LLVMPasses and LLVMCGData: no instructions:u significance - I don’t expect LLVMPasses to be hot anyway. On the code size metric, both libLLVMPasses.a and libLLVMCGData.a become slightly smaller

LLVM Compile-Time Tracker

the migrated objects shrink — PassBuilderPipelines.cpp.o −75.8 KB (−7.9%), StableFunctionMap.cpp.o −22.5 KB (−26.8%), CodeGenData.cpp.o −9.2 KB, CodeGenDataReader.cpp.o −6.1 KB; libLLVMPasses.a −35.8 KB, libLLVMCGData.a −11.3 KB — against a one-time cost in the infrastructure: OptTable.cpp.o +9.2 KB (`applyArgs`, `printHelpOptions`), CommandLine.cpp.o +7.2 KB (the dispatch), OptionsRegistry.cpp.o new (libLLVMSupport.a +17.4 KB), and OptionParserEmitter.cpp.o +28.7 KB in llvm-tblgen only; every tool's generated Options.inc is flat. PassesOptions.cpp.o for the 33 options is 13.6 KB (a 98-row, 24-byte table, 3.9 KB of strings, the `apply` switch) plus a 195-byte static initializer; the `cl::opt`s it replaces were 6.6 KB of static-initialization code, 7.2 KB of `.bss` and 52 relocations.

It would be good if “Current” would be a different name that is more easily and unambiguously greppable, as such uses indicate global option access that we want to phase out.

Renamed Current to Global: PassesOptions::Global.RunNewGVN . Updated the post as well.

We could alternatively also stop adding plugin options to the global option namespace… to me this always seemd to be a weird design that was imposed by llvm::cl only supporting such use. While I added this code to support options in plugins as the easiest way to do so, we could also introduce some -plugin-arg or (probably more preferable?) go for some syntax like -load-pass-plugin=foo.so,-opt,-no-foo. I wouldn’t worry too much about supporting option registration while parsing options.

Agreed. We can introduce sth like -plugin-arg when we come to migrate {opt,llc}.

Not sure I like this bit, this would make it much harder to change the representation. Support options could also be moved elsewhere (I don’t think TableGen depends on Support parts that have options). For TableGen we could also hand-roll a very minimal option parser.

llvm-min-tblgen depends on Support. Support has a few cl::opt option, which block the migration to TableGen. We could move these options to a library above Support.
To understand better, one must put it into practice.

Given it’s uses in clang-tools-extra, polly, Flang, MLIR, Clang, Bolt, it’ll be a long time until we can consider to actually remove llvm::cl.

Yes, even porting llvm/ alone (~70 libraries) would take a considerable amount of time.

When we eventually remove llvm::cl, we will swap out llvm::cl::ParseCommandLineOptions for a custom parser that copies its behavior.

[rnk]

Thanks for working on this! I think the most important thing here for most consumers (NVIDIA included) is the removal of global state.

This is one of many in a long line of global state removal projects, such as the one which introduced LLVMContext, which I think is when we got rid of global Type and Constant singletons.

I think this is one of those tech debt paydowns that would never make it to the top of the priority heap without AI tools. Please add me as a reviewer for these PRs.

Something to consider: How do you force link the option table registration? If those are standalone unreferenced cpp files, the static linker will drop them. You need the registration root to ride along in some core cpp file which provides symbols that library clients use.

[MaskRay]

Thanks! Will add you.

@rnk

Something to consider: How do you force link the option table registration? If those are standalone unreferenced cpp files, the static linker will drop them. You need the registration root to ride along in some core cpp file which provides symbols that library clients use.

Take LLVMCGData’s CGDataOptions as an example. Every way of reaching CGDataOptions goes through a symbol that is only defined in CGDataOptions.cpp.o. The object-file-format-neutral linker rule (COFF/ELF/Mach-O/XCOFF/…) ensures that the archive member CGDataOptions.cpp.o within libLLVMCGData.a will be loaded/extracted.

• (global instance) CGDataOptions::Global.X
• Ctx.getOptions<CGDataOptions>() references return *static_cast<const T *>(Slots[CGDataOptions::Slot].get());

// llvm/lib/CGData/CGDataOptions.cpp
// license banner ...
#include "CGDataOptions.h"
#include "llvm/Option/LibraryOptions.h"

#define OPTIONS_STRUCT_DEFS
#include "CGDataOptions.inc"

static llvm::opt::RegisterLibraryOptions<llvm::CGDataOptions> Registration;

[MaskRay]

Tangentially related:

Cl::opt<bool> spellings: accept -no-<name>? retire -<name>={0,1,false,true}?

Should cl::opt<bool> accept -no-<name>/--no-<name>? Should we eventually reject -<name>=0/--<name>=0 and -<name>=false/--<name>=false (and their =1 =true variants)? 1. Accept -no- [Support] Accept -no-<name> to negate cl::opt<bool> by MaskRay · Pull Request #224978 · llvm/llvm-project · GitHub makes -no-<name> set a cl::opt<bool> or cl::opt<cl::boolOrDefault> to false. -no-x is how compiler drivers (-fno-), linkers/binutils (--no-), git, curl, V8, Python argparse and the majority of command …

If options move to TableGen one library at a time, =false/=0 could be rejected one library at a time, giving users on rolling updates (many corporate users of llvm-project) time to adapt.

It’s the best time to retire =false/=0 for cl::opt<bool>, if we want to retire them at all.

[danilaml]

Not sure what rejecting =0/1 and =false/true would buy us other than enforcing certain style (but somewhat losing the consistency of -option=value)

[MaskRay]

Progress update.

Landed: OptTable size, making it a cheaper option representation (#224374, #224508, #224805, #224807, #224813, #224818, #224822)

• Info is 24 bytes, down from 88.
• The strings are string table offsets.
• TableGen emits the tables in a function with no relocations.

Landed: cl:: features removed or restricted

• #224953 Remove cl::Sink
• #224959 Remove cl::multi_val
• #224975 Accept only true, false, 1 and 0 for cl::opt<bool> (-flag= is now an error)
• #225609 Remove cl::DefaultOption (-h is now --help)
• #225870, #225628 Remove unneeded cl::Optional and cl::ZeroOrMore from named options. NFC
• #224958 Remove cl::Grouping

Landed: tools moved to OptTable, so that the parser modes above no longer have users

• #224955 llvm-cov gcov (the last explicit cl::Grouping user)
• #224964 yaml2obj, #224965 obj2yaml, #224966 split-file (the last LongOptionsUseDoubleDash users)

In review

• #226347 Remove LongOptionsUseDoubleDash from cl::ParseCommandLineOptions
• #226359 [BOLT] Remove cl::callback uses
• #226087 [CGData] Declare command line options in TableGen, one struct per library
  • This is the first library to migrate. It adds the TableGen support (OptionsStruct, BoolField, ValueField), cl::LibraryOptions for cl:: dispatch, and -help-hidden output.
  • Migrating a library takes one .td file, a small private header, and one static opt::RegisterLibraryOptions<T> line.
  • T::Global is constant-initialized: string members are StringRef, and the parser keeps its own copy of each value.
  • Bools accept -x, -no-x and -x=true|false|1|0, as cl::opt<bool> does today.
• #224978 Accept -no-<name> for cl::opt<bool>. Parked, since the replies in Cl::opt<bool> spellings: accept -no-<name>? retire -<name>={0,1,false,true}? were split on -no-x.

Next

• The remaining cl::callback users: 10 in libraries.
  • Two need a hook that runs after parsing: -print-debug-counter and -time-passes-per-run.
  • GISel’s -disable-rule and -only-enable-rule can merge by position.
• std::optional<T> fields, for options whose code checks getNumOccurrences().
• Enum fields, then LLVMPasses.

[MaskRay]

Withdrawing -no-<name>. Given the split replies in the spelling thread, I’m dropping -no- for boolean options. I’ll close #224978, and the migrated options no longer accept -no-x. A BoolField accepts -x and -x=true|false|1|0, exactly what cl::opt<bool> accepts today.

MLIR
To check that the design works outside llvm/, I ported MLIRContextOptions (--mlir-disable-threading, --mlir-print-op-on-diagnostic, --mlir-print-stacktrace-on-diagnostic) on top of #226087. The port adds a 22-line .td and removes 34 lines from MLIRContext.cpp: the ManagedStatic, and every clOptions.isConstructed() check.

• registerMLIRContextCLOptions() registers the struct from a function-local static, so MLIR’s -Werror=global-constructors still holds. MLIRContextImpl’s members default to MLIRContextOptions::Global’s values, which are the old defaults when a tool never registers.

• A read of --mlir-disable-threading becomes one compare. Before, it was three loads of the ManagedStatic pointer, two null tests, and a byte compare, plus a cold RegisterManagedStatic call in each reader:

  leaq  mlir::MLIRContextOptions::Global(%rip), %rax
  cmpb  $0x0, (%rax)
  

  The MLIRContext constructor goes from 355 to 298 instructions.

• mlir-opt --help and --help-hidden list the same options as before, now under “Library options:”. The MLIR lit tests that use these options pass.

MLIR still needs enum and list fields, std::optional fields for options whose readers check getNumOccurrences(), cl::cat-style groups, and a flag that implies a default value (--split-input-file). Each can be added without changing the interface. Pass options, which are parsed per instance from pipeline strings, are out of scope.

Next

• The remaining cl::callback users. -load-pass-plugin loads plugins from a callback during parsing, so it moves to two-phase parsing: collect the plugin paths, load them, then parse the rest.
• std::optional and enum fields, then LLVMPasses.

[nikic]

Thanks for working on this! I expect there will be details to sort out along the way, but your high level plan looks very reasonable to me.

[MaskRay]

Thanks, nikic. #226087, the framework change, was approved by @aengelke. I plan to merge it in 48h, unless there are objections before then.

Since the last update:

• Struct members are snake_case and derived from the spelling: -codegen-data-generate sets codegen_data_generate, matching OptTable’s OPT_codegen_data_generate. A named defm overrides this. This allows rg codegen.data.generate to find both option spelling and uses.
• Library options are hidden by default, since most are developer knobs. -help-hidden lists them; let Hidden = 0 in { ... } (which will be used by llvm/tools/*/) lists them in -help.

A few things are left for future:

• -print-options, “Did you mean” suggestions and cl::OptionCategory don’t cover library options yet.
• @aengelke suggested making structs public for LLVM-as-a-library users who want to set values without building argv strings. Structs stay private by default; a library can expose its struct when a user needs it (should use a proper API struct like PipelineTuningOptions/TargetOptions/lto::Config)
• Also from @aengelke: classifying options (debug, tuning knob, migration toggle) for grouping in -help.
• Also from @aengelke: copying only the argument values the struct keeps, using BumpPtrAllocator

On pass plugins: #226773 loads -load-pass-plugin plugins after cl::ParseCommandLineOptions, instead of from a callback during parsing. A plugin gets its arguments explicitly through -plugin-arg=<plugin>,<arg> (similar to clang -fplugin-arg). A loaded plugin’s options are therefore no longer accepted as ordinary options. For Polly, opt accepts the same spelling whether Polly is linked in or loaded as a plugin, e.g. opt -plugin-arg=Polly,-polly-tile-sizes=1,64. This is a user-visible change for plugin users, so comments on #226773 are welcome.

---

MLIR’s cl::opt has a migration path as well. The way it registers library options is better than LLVM’s, and the framework improves it further. The trouble is pass options and registry-driven flags, which use cl:: as a general-purpose string parser. Those parts constrain the cl:: cleanup more than anything in LLVM, but they don’t block llvm/ library migration.

Updated the initial post.

[jyknight]

@MaskRay

• Library options are hidden by default, since most are developer knobs. -help-hidden lists them; let Hidden = 0 in { ... } (which will be used by llvm/tools/*/) lists them in -help.

I’m a bit confused by this: do we want to make migrated options more-hidden by default than they are with today’s cl::opt? I note we already have cl::Hidden scattered randomly around, there, not sure if that is done with any useful intent or not.

Isn’t this entire options parser intended for developer knobs, with user-facing tools using their separate OptTable parser?

[MaskRay]

I’ve dropped let Hidden = 0 in { ... } from [Option] Declare library command line options in TableGen by MaskRay · Pull Request #226087 · llvm/llvm-project · GitHub . Library options are hidden and only listed in -help-hidden. In general tool-specific options are non-hidden. In the remote end state llvm-mc/llc/opt should use OptTable instead of library options.

Quick survey: 2,589 of the 2,939 cl::opts in llvm/lib (88%) are already cl::Hidden.

[nickdesaulniers]

Maybe unrelated, but my biggest complaint wrt. cl::opt is that half of these would be dropped during LTO. There were a couple I converted to IR Metadata to avoid this, but there are way way more flags than is feasible to convert by hand, and probably don’t make sense to do so.

[MaskRay]

Bitcode is backward compatible, while library options (developer knobs) can be added, modified, removed at any time (though in practice we would not remove frequently used options like -x86-asm-syntax, -import-instr-limit). A setting that survives LTO should be an IR attribute or module flags metadata - these have non-trivial boilerplate cost, so we the bar adding an IR thing is relatively high.

This RFC doesn’t change that, but having each library’s options in one struct makes it easier to spot the ones that should become IR, or moved to a proper struct intended to be API. This is also a reason XXXOptions.td files are placed in llvm/lib/XXX/ instead of llvm/include/llvm/XXX/.