[RFC][TableGen] Add let append/prepend syntax for field concatenation

Motivation

LLVM TableGen currently lacks a way to accumulate field values across class hierarchies. When a derived class sets a field via let, it completely replaces the parent’s value. This forces users into verbose workarounds like (example edited from the original not functional one):

class Op { // This is generic MLIR Base 
  code extraClassDeclaration = ?;
}

// Some Generic shared base
class MyShared1OpClass : Op {
  code shared1ExtraClassDeclaration = [{ some generic code 1 }];
}

class MyShared2OpClass : MyShared1OpClass {
  code shared2ExtraClassDeclaration = [{ some generic code 2 }];
}

def MyOp : MyShared2OpClass {
  // need to manually concatenate shared code
  let extraClassDeclaration =   
      shared1ExtraClassDeclaration
    # shared2ExtraClassDeclaration
    # [{ additional specialized code }]; 
}

Instead I propose a more natural incremental solution without unnecessery intermediate definitions:

class Op {
  code extraClassDeclaration = ?;
}

class MyShared1OpClass : Op {
  let append extraClassDeclaration = [{ some generic code 1 }];
}

class MyShared2OpClass : MyShared1OpClass {
  let append extraClassDeclaration = [{ some generic code 2 }];
}

def MyOp : MyShared2OpClass {
  let append extraClassDeclaration = [{ additional specialized code }]; 
}

This is especially painful in MLIR, where dialect authors want base op/type/attribute classes to inject shared C++ declarations into all derived definitions. I attempted to solve this in PR [MLIR][TableGen] Add inheritableExtraClassDeclaration/Definition for Op and AttrOrTypeDef by xlauko · Pull Request #182265 · llvm/llvm-project · GitHub with MLIR-specific inheritableExtraClassDeclaration/Definition fields, but as @mehdi_amini pointed out, this is ad-hoc – the same inheritance problem exists for traits, arguments, results, and any other list/string/dag field. Rather than adding inheritable* variants per field, we should solve this at the language level.

Design

This PR adds two new modifiers to the let statement: append and prepend.

class Base {
  list<int> items = [1, 2];
  string text = "hello";
  dag d = (op);
}

def Example : Base {
  let append items = [3, 4];    // items = [1, 2, 3, 4]
  let prepend items = [0];      // items = [0, 1, 2]
  let append text = " world";   // text = "hello world"
  let prepend text = "say ";    // text = "say hello"
  let append d = (op 3:$a);     // d = (op 3:$a)
}

Supported types

Field type Operation Concat operator
list<T> append/prepend !listconcat
string / code append/prepend !strconcat
dag append/prepend !con
Other (bit, int, bits) Error

Semantics

  • let append concatenates the new value after the current value
  • let prepend concatenates the new value before the current value
  • If the current value is unset (?), the new value is used directly
  • A plain let (without modifier) still replaces, allowing opt-out from accumulated values
  • Works in both body-level (def Foo { let append ... }) and top-level (let append ... in { }) contexts

Multi-level inheritance

Accumulation works naturally across inheritance chains:

class Base {
  list<int> items = [1, 2];
}

class Middle : Base {
  let append items = [3];    // items = [1, 2, 3]
}

def Leaf : Middle {
  let append items = [4];    // items = [1, 2, 3, 4]
}

Multiple inheritance

TableGen supports multiple inheritance (def D : A, B { ... }), where parent classes are processed left to right and the last parent class’s value wins for any shared field. let append/let prepend operates on whatever value the field has after inheritance resolution — it does not accumulate across sibling parents:

class A { list<int> items = [1, 2]; }
class B { list<int> items = [3, 4]; }

def D : A, B {
  let append items = [5];  // items = [3, 4, 5]  (A's value is lost)
}

This also applies to diamond inheritance:

class Base  { list<int> items = [1]; }
class Left  : Base { let append items = [2]; }  // [1, 2]
class Right : Base { let append items = [3]; }  // [1, 3]

def D : Left, Right {
  let append items = [4];  // items = [1, 3, 4]  (Left's [2] is lost)
}

This is consistent with how plain let works with multiple inheritance — it is the standard last-writer-wins rule. Users who need accumulation from multiple parents should use a single-inheritance chain instead.

Backward compatibility

This proposal is fully backward compatible. The keywords append and prepend are implemented as context-sensitive keywords — they are only recognized as modifiers when they appear immediately after let (in both body-level and top-level contexts). In all other positions, append and prepend remain valid identifiers and can be used as field names, class names, def names, etc. This means:

  • No existing .td files (in-tree or out-of-tree) will break
  • Fields named append or prepend continue to work: let append append = [5]; is valid (the first append is the modifier, the second is the field name)
  • The parser checks for the identifier string value after let, not for a reserved token

Implementation in PR [TableGen] Add let append/prepend syntax for field concatenation by xlauko · Pull Request #182382 · llvm/llvm-project · GitHub

7 Likes

Maybe I’m missing something, but this does not compile at the moment (Compiler Explorer). Is this just an illustrative example of the sort of thing done with MLIR’s classes, but shown with the built in types?

Could you show one that’s in use today in MLIR? Or is your point that you can’t even write a valid workaround because of TableGen’s limitations?

I thought about how you could do this with a !super() instead, and not have to write the parent class name all the time. This is easier to get right as an author, but no less verbose really. Plus, !super() would have all sorts of other uses that make it a lot more complex. Basically the reverse of how you explained your idea would work with multiple inheritance and so on.

Then I wondered if you even need the parent class name, and this does compile:

class Base {
  list<int> items = [1, 2];
}

class Derived : Base {
  // Must manually repeat parent values:
  let items = !listconcat(items, [3, 4]);
}

But fails as soon as you make a def:

def ADerived: Derived {}
<source>:10:5: error: Initializer of 'items' in 'ADerived' could not be fully resolved: !listconcat(!listconcat(items, [3, 4]), [3, 4])

So you can’t do the equivalent of Python self.alist = self.alist + [3, 4].

Oh, I guess my earlier example was oversimplified to the point of being non-compilable. I fixed the exmaple. But to clarify, the actual issue I was trying to solve in MLIR is this:

class Op { // This is generic MLIR Base 
  code extraClassDeclaration = ?;
}

// Some Generic shared base
class MySharedOpClass : Op { 
  code sharedExtraClassDeclaration = [{ some generic code }];
}

def MyOp : MySharedOpClass {
  // need to manually concatenate shared code
  let extraClassDeclaration = sharedExtraClassDeclaration 
    # [{ additional specialized code }]; 
}

You can alternatively passthrough extraClassDeclaration as parameter to MySharedOpClass

class Op {
  code extraClassDeclaration = ?;
}

class MyOpClass<code appendExtraClassDeclaration> : Op {
  let extraClassDeclaration = [{ some generic code }]
     # appendExtraClassDeclaration;
}

def MyOp : MyOpClass<[{ additional specialized code }]>;

However, this approach quickly becomes unwieldy when multiple parameters must be composed this way. It’s also problematic when the base class defines a value that should be extended rather than overwritten.

With the proposed change, this becomes much more natural:

class Op { // This is generic MLIR Base 
  code extraClassDeclaration = ?;
}

// Some Generic shared base
class MySharedOpClass : Op { 
  let extraClassDeclaration = [{ some generic code }];
}

def MyOp : MySharedOpClass {
  let append extraClassDeclaration = [{ additional specialized code }]; 
}

Thanks, understand the use case better now.

If I write your MLIR themed example using the verbose workaround shown in the first post, it looks like this:

class Op { // This is generic MLIR Base 
  code extraClassDeclaration = ?;
}

// Some Generic shared base
class MySharedOpClass { 
  code extraClassDeclaration = "shared things";
}

def MyOp : MySharedOpClass {
  code extraClassDeclaration = !strconcat("extra ", extraClassDeclaration); 
}

This does not compile, because:

<source>:10:5: error: Initializer of 'extraClassDeclaration' in 'MyOp' could not be fully resolved: !strconcat("extra ", !strconcat("extra ", extraClassDeclaration))

But it feels like it’s equivalent to your let append. Is it, and does that mean you have solved this same issue in your implementation of let append?

I’m probably missing a detail there but my point is that if they are equivalent then -

The proposed change is much more natural than the MLIR themed example you gave, but the gap is much smaller if you assume that the verbose workaround could be made to work.

let append a = "abc"; vs. let a = !strconcat(a, "abc");

Despite that, I do think let append has advantages:

  • I don’t need to repeat a.
  • I don’t need to figure out which concat operator to use.
  • Lower risk of side effects from making the member name valid in the assignment. let foo = foo; doesn’t turn from an error (error: Recursion / self-assignment forbidden) into silently doing nothing.

I just needed to set my expectations for verbosity, as the first workaround shown didn’t seem that verbose in the grand scheme of things.

This example does not work so I think saying it’s a workaround is slightly confusing. That being said, perhaps you could use this example to illustrate what this RFC tries to achieve. That is, “I propose a new syntax so that people can essentially do what this snippet tries to achieve”

Personally I like the !super idea mentioned by @DavidSpickett a little more. I prefer not to add a new programming language keyword unless it’s absolutely necessary.

As a side-effect this might help to clean up other MLIR classes, e.g. handling of traits. Traits flow backwards: the leaf def passes traits as a parameter up through each ancestor, which threads them forward via !listconcat.

// Level 1 — base arithmetic class
// traits=[] exists only to receive traits from descendants
class LLVM_ArithmeticOpBase<Type type, string mnemonic,
                            string instName, list<Trait> traits = []> :
      LLVM_Op<mnemonic,
              !listconcat([SameOperandsAndResultType, NoMemoryEffect], traits)> {
    // ...
}

// Level 2 — adds overflow interface, passes the rest through
// traits=[] exists only to receive traits from descendants
class LLVM_IntArithmeticOpWithOverflowFlag<string mnemonic, string instName,
                                           list<Trait> traits = []> :
      LLVM_ArithmeticOpBase<AnySignlessInteger, mnemonic, instName,
         !listconcat([DeclareOpInterfaceMethods<IntegerOverflowFlagsInterface>],
                      traits)> {   // ← injects its trait, then passes traits up
  // ...
}

// Leaf — [Commutative, Pure] travel *up* two levels before landing
def LLVM_AddOp : LLVM_IntArithmeticOpWithOverflowFlag<"add", "Add",    
    [Commutative, Pure]>;

To trace what LLVM_AddOp actually gets you have to mentally unwind:

  1. [Commutative, Pure] passed into IntArithmeticOpWithOverflowFlag
  2. concatenated with [DeclareOpInterfaceMethods], passed into ArithmeticOpBase
  3. concatenated with [SameOperandsAndResultType, NoMemoryEffect], passed into LLVM_Op.

The same traits = [] boilerplate repeats identically on every intermediate class in the file (LLVM_TerminatorOp, LLVM_IntArithmeticOp, LLVM_IntArithmeticOpWithExactFlag, LLVM_IntArithmeticOpWithDisjointFlag, LLVM_FloatArithmeticOp).

After: let append traits.

Each level owns what it adds. Traits flow forward, downward:


  // Level 1 — declares what it owns, no traits parameter
  class LLVM_ArithmeticOpBase<Type type, string mnemonic, string instName> :
      LLVM_Op<mnemonic, [SameOperandsAndResultType, NoMemoryEffect]> {
    // ...
  }

  // Level 2 — appends what it adds, no traits parameter
  class LLVM_IntArithmeticOpWithOverflowFlag<string mnemonic, string instName> :
      LLVM_ArithmeticOpBase<AnySignlessInteger, mnemonic, instName> {
    let append traits = [DeclareOpInterfaceMethods<IntegerOverflowFlagsInterface>];
    // ...
  }

  // Leaf — appends what it adds
  def LLVM_AddOp : LLVM_IntArithmeticOpWithOverflowFlag<"add", "Add"> {
    let append traits = [Commutative, Pure];
  }
1 Like

This does only adds local modifier to let statement not a new keyword.

The proposed change works also nicely in other places of the language like:

  class Instruction {
    list<string> Predicates = [];
  }

  let append Predicates = ["HasSSE2"] in {
    let append Predicates = ["Is64Bit"] in {
      def ADD64 : Instruction;
      // Predicates = ["HasSSE2", "Is64Bit"]
    }
    def ADD32 : Instruction;
    // Predicates = ["HasSSE2"]
  }

Where !super would not help, or at least sounds unintuitive?

I should be clear that by this:

Plus, !super() would have all sorts of other uses that make it a lot more complex. Basically the reverse of how you explained your idea would work with multiple inheritance and so on.

I meant the downside is that it has more flexibility that we have to define the semantics of, or limit its use and end up with something that is surprisingly limited compared to what I’d expect if I knew super from other languages.

We could come up with some other name for it, !self perhaps, but then we’ve got an operator that only works at a specific point in the program and potentially situations where foo and !self().foo refer to the same thing and I’d rather have one way to do things.

Yes, using !super here, if you even could, would be “inside out” I think.

With let you’re surrounding defs with the adjustment you want to make, with a !super you’d use it within classes. Same with the hypothetical !concat workaround, that has to be done in the def or class.

I get the feeling that let append and !super are in two different arenas:

  • let append is in the data oriented, more functional side of things.
  • !super is in the OOP side.

And while Tablegen has inherited bits of both styles, the approach your example takes:

let any_member_with_this_name_actually_be = a_value in {
  // loads of defs
}

Is to me more iconic TableGen use than the OOP stuff (I know “iconic” is subjective, idiomatic? appealing? can’t think of a more scientific word).

Another verbosity we could consider is how many pages of docs do I need to read to understand a thing and how many questions does it raise.

When I read let append I’m thinking:

  • I take the some_member I already have, and append to it. It’s self.foo = self.foo + bar in Python terms.
  • This is not quite “reading the code explains the code” entirely, but it’s pretty close (you need to realise that self.foo already exists`).

For !super I’m thinking:

  • I look to the parent (what is the parent?) and set my some_member to their some_member plus something.
  • Now I need to read about which parent is chosen.

So if 99% of the use cases are self.foo = self.foo + bar, why worry people with what the parent class is?

And I was going to say, one thing !super could allow is to get things that are only a member of the parent but not the current class. Except - TableGen doesn’t have public and private class members and everything is inherited.

So unless you want to go back through multiple parents, there’s no advantage to !super.

Overall I’m against my !super suggestion.

I’ve written a lot so I will summarise:

  • The verbose workaround(s) are verbose and require thinking about what the parent class is and which concat operators to use. They are too verbose to be easily maintained.
  • The hypothetical less verbose workaround requires compiler changes too, is still more verbose than let append, and you have to find the right concat operator to use.
  • Compared to !super, let append does not require users to think about what the result of super is, they can just think within the context of the current class.
  • It composes well with let ... in { } which I think is a trademark / important / common pattern in TableGen.
  • !super could do many more things, but would be more complex as a result and attempting to constrain it could produce something less verbose by code word count, but way more verbose because you’ve got to read more docs to understand it.
  • !super feels like an OOP focused feature that if it were to exist, needs to be justified with a use case that requires all of it’s potential complexity.

That said, I’m just playing at language designer here so I’m interested in what @mshockwave thinks to all that.

If we were to do things other than append or prepend, we’d either have way more keywords (albeit only in let ___ varname, or we need some form of self reference to allow things like:

class Base {
  string name = "lower_case";
}

class Derived {
  string name = !toupper(name);
}

def ADerived: Derived {}
<source>:9:5: error: Initializer of 'name' in 'ADerived' could not be fully resolved: !toupper(!toupper(name))

(Compiler Explorer)

(though I realise this proposal itself does not want to do arbitrary things, and sometimes we trade convenience for simplicity)

Perhaps that is part of the concern about keywords?

I’m in favour of this proposal, with the new keywords and the described semantics.

In the RISC-V backend, we frequently want to add to Predicates, which are usually defined in an outer let, and right now the semantics of let mean we cannot nest these, because the inner let would fully override the outer let. We’ve had a few bugs caused by this, IMO partly because we would like a nesting structure, but it doesn’t yet exist. Adding let append or let prepend would give us the nicer nesting structure.

I’m not really in favour of any form of super because we already have multiple inheritance, and TableGen’s inheritance is not like regular OO inheritance. To me it does seem like super is a far larger feature than just let append and let prepend.

2 Likes

Sorry for the earlier confusion with the broken example. @mshockwave @DavidSpickett do you have any objections to moving forward with the proposal as it stands? My thinking is that implementing full TableGen inheritance with !super would require a much deeper design discussion to make it work consistently across other use cases. For this particular use case, it’s still the more verbose option and doesn’t interact as cleanly with outer let bindings.

1 Like

My only reservation is that if you want anything other than prepend or append, you’re adding more keywords each time. For example adding or subtracting from a number.

(though you could use the length of a list as the final number, but I digress)

Then again, no one’s actually asking for that. They’re asking for append and prepend :slight_smile: . And I don’t have much experience in how to weigh hypothetical future concerns.

My line of thinking is something like this, starting with the proposed usage:

def MyOp : MyShared2OpClass {
  let append extraClassDeclaration = [{ additional specialized code }]; 
}

If we allowed arbitrary operators, it would be like saying:

def MyOp : MyShared2OpClass {
  let !strconcat extraClassDeclaration = [{ additional specialized code }]; 
}

(which we could do, but it’s only one operator, what about complex combinations etc etc)

Then, if we were allowed to reference the name after = we could write instead:

def MyOp : MyShared2OpClass {
  let extraClassDeclaration = !strconcat(extraClassDefinition, [{ additional specialized code }]); 
}

“If we were allowed” could be a tiny change, could be a massive rewrite. I’m not going to ask anyone else to pursue it.

Perhaps there is a more general idea to be found, but I don’t have a proposal that has a known scope. If one appears in future and it supersedes let append then we have the freedom to pick one, use both, whatever we need to do as the source of truth on TableGen.

So I am in favour of let append.

If this was only let append, I would think about calling it let concat to match the naming of the ! operators.

I can’t think of an intuitive name for let prepend though. append is !con(a, b) and prepend is !con(b, a). They’re both concatinations.

Also if we overlap the names too much it kinda makes it look like the syntax is let <drop the !><operator name>, which is not actually the case.

So stick to append and prepend.

1 Like

My concern was primarily on adding new language syntax while (I thought) we can just do it with a new bang operator (i.e. !super). This is more of my philosophical standpoint: I’ve seen countless of programming languages that “started small” (in terms of number of language features) eventually evolved into a behemoth with a gazillion features to a point where no one has the full picture of why they were invented in the first place. I simply don’t want TableGen to become that.

That being said, I am now convinced that there are probably more works to support !super , and allowing self-referencing – which was also discussed in this thread – might be too verbose. More broadly speaking, this is a pattern that has been used in many places so perhaps we should optimize for it. Therefore, I’m also in favor of let append & let prepend.

5 Likes