TableGen and Greenspun

The TableGen language seems to be growing Lisp macros from two different directions.

Che-Liang Chiou added a preprocessor with for loops, and David Greene added multidefs.

It seems that some kind of macro facility is needed, perhaps we should discuss what it is supposed to look like?

/jakob

Jakob Stoklund Olesen <jolesen@apple.com> writes:

The TableGen language seems to be growing Lisp macros from two
different directions.

Che-Liang Chiou added a preprocessor with for loops, and David Greene
added multidefs.

It seems that some kind of macro facility is needed, perhaps we should
discuss what it is supposed to look like?

Don't throw the Greenspunning term around unless you really mean it.

The problem I solved via multidefs was this: how does one write a set of
Pat<> patterns in a generic way?

For example, I want to be able to do this:

defm MOVH :
  vs1x_fps_binary_vv_node_rmonly<
    0x16, "movh", undef, 0,
           // rr
           [(undef)],
           // rm
           [(set DSTREGCLASS:$dst,
                 (DSTTYPE (movlhps SRCREGCLASS:$src1,
                                 (DSTTYPE (bitconvert
                                             (v2f64 (scalar_to_vector
                                                       (loadf64 addr:$src2))))))))],
           // rr Pat
           [],
           // rm Pat
           [[(DSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
             (MNEMONIC SRCREGCLASS:$src1, addr:$src2)],
            [(INTDSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
             (MNEMONIC SRCREGCLASS:$src1, addr:$src2)]]>;

I need to have the various types substituted and the mnemonic with
appropriate suffixes substituted. I need to have this done for each
Pat<> pattern in the list ("rm Pat" above). To do this TableGen needs
to know something about the top-level defm (the "defm name," if you
will) and something about the types and classes the defm gets
instantiated with.

Therefore, I need some way to process a list of patterns and do
substitutions in the context of a specific defm and all of the classes
it inherits from.

I'm open to another solution but multidef has served its purpose well
and I would need equivalent functionality.

TableGen is missing a lot of functionality but I think it should be
added incrementally rather than rewritten in some grand scheme.

                               -Dave

greened@obbligato.org (David A. Greene) writes:

The problem I solved via multidefs was this: how does one write a set of
Pat<> patterns in a generic way?

For example, I want to be able to do this:

defm MOVH :
  vs1x_fps_binary_vv_node_rmonly<
    0x16, "movh", undef, 0,
           // rr
           [(undef)],
           // rm
           [(set DSTREGCLASS:$dst,
                 (DSTTYPE (movlhps SRCREGCLASS:$src1,
                                 (DSTTYPE (bitconvert
                                             (v2f64 (scalar_to_vector
                                                       (loadf64 addr:$src2))))))))],
           // rr Pat
           [],
           // rm Pat
           [[(DSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
             (MNEMONIC SRCREGCLASS:$src1, addr:$src2)],
            [(INTDSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
             (MNEMONIC SRCREGCLASS:$src1, addr:$src2)]]>;

I need to have the various types substituted and the mnemonic with
appropriate suffixes substituted. I need to have this done for each
Pat<> pattern in the list ("rm Pat" above). To do this TableGen needs
to know something about the top-level defm (the "defm name," if you
will) and something about the types and classes the defm gets
instantiated with.

Therefore, I need some way to process a list of patterns and do
substitutions in the context of a specific defm and all of the classes
it inherits from.

So here's why I think the for loop proposal can't be a preprocessing
phase. Down in the guts of this I fundamentally need to be able to do
this:

multiclass blah<list<int> Values> {
  for v = Values {
    def DEF#v : base_class<v>;
  }
}

Will that work? Is the for loop evaluated after parameter binding?

I like the for loop idea. But it can't be "just" a preprocessing step.

Also, I know I introduced the #..# "pasting" operation but I've found it
to be too limiting. In this example:

(Equivalent TableGen code with a for-loop)

greened@obbligato.org (David A. Greene) writes:

Also, I know I introduced the #..# "pasting" operation but I've found it
to be too limiting. In this example:

[snip!]

what if we instead did this:

(Equivalent TableGen code with a for-loop)
----------------------------------------
multiclass PTX_FLOAT_4OP<string opcstr, SDNode opnode1, SDNode opnode2> {
  for nbit = [32, 32, 64, 64],
      op_suffix = [r, i, r, i],
      op_type = [RegF32, f32imm, RegF64, f64imm],
      op_node_type = [RegF32, fpimm, RegF64, fpimm] in {
    def !strconcat(!strconcat("rr", !cast<string>(op_suffix)), "nbit")
      [...]
  }
}

Yes, it's a bit more verbose but also more flexible in what you can do
with iterator values.

Here's some more detail on this.

I want to be able to use #..# pasted stuff in a variety of contexts.
For example:

multiclass blah<string intr> {
  for prefix = [<empty>, Vx, Vy] {
    for suffix = [ps, pd] {
      def #prefix#ADD#suffix# : ProcessSomePattern<
        [(set RegClass:$dst, (!cast<Intrinsic>(#prefix#intr#suffix)
                              RegClass:$src1, RegClass:$src2))]
    }
  }
}

It's actually more complex than this but this gives the basic idea. The
key thing I need to be able to do is construct an instrinsic name given
some base string, a prefix and a suffix.

Here's a critical question. What type do the for loop iterators have?
string?

If the iterators are strings, I know that this will work:

!cast<Instrinsic>(!strconcat(!strconcat(prefix, intr), suffix))

Will this work as well?

!cast<Intrinsic>(#prefix#intr#suffix)

If so, I think I can get away with using #..#. If for loop iterators
are typed as strings that would also allow me to do what I need via
strconcat.

Actually, having both iterators as strings and paste "operator" results
as strings would be useful.

                                -Dave

greened@obbligato.org (David A. Greene) writes:

So here's why I think the for loop proposal can't be a preprocessing
phase. Down in the guts of this I fundamentally need to be able to do
this:

multiclass blah<list<int> Values> {
  for v = Values {
    def DEF#v : base_class<v>;
  }
}

Will that work? Is the for loop evaluated after parameter binding?

I think I can actually make this work using the existing multidef
feature code. The ideas are fundamentally the same. I do like the
for syntax a lot better.

Che-Liang, would you mind if I played around with changing the multidef
parsing to recognize your for syntax? Your code will work in places
that multidefs can't right now (as Jakob noted) and we want to retain
that ability. I would be working on integrating for loops into TableGen
proper, not as a preprocessor.

                                -Dave

I want to be able to use #..# pasted stuff in a variety of contexts.
For example:

multiclass blah<string intr> {
for prefix = [<empty>, Vx, Vy] {
   for suffix = [ps, pd] {
     def #prefix#ADD#suffix# : ProcessSomePattern<
       [(set RegClass:$dst, (!cast<Intrinsic>(#prefix#intr#suffix)
                             RegClass:$src1, RegClass:$src2))]
   }
}
}

It's actually more complex than this but this gives the basic idea. The
key thing I need to be able to do is construct an instrinsic name given
some base string, a prefix and a suffix.

David, I think the syntax you are proposing here is much better than the multidef syntax.

When I look at test/TableGen/MultiPat.td, I have a really hard time figuring out what is happening. Basically, I don't want our target descriptions to end up looking like that.

Here's a critical question. What type do the for loop iterators have?
string?

That's probably good enough.

/jakob

This kind of thing is very hard to read and understand.

ISAs don't change a lot, and .td files are read lot more than they are written. I actually think it is preferable to leave some redundancy in the .td files, it makes it much easier to find things.

It is already tedious to expand multiclasses in your head when you are looking for the properties of a specific instruction.

/jakob

+1 for not adding a preprocessor.

There seems to be two use cases for this:

1. Generating a sequence of similar top-level defs, for example registers named R0--R15.

2. Injecting patterns into multiclasses.

It seems to me that the same mechanism could support both.

/jakob

This is a critical point. The readability of the .td files directly relates to their maintainability, and the ease of access for newer contributors.

-Jim

Jakob Stoklund Olesen <jolesen@apple.com> writes:

Che-Liang., would you mind if I played around with changing the multidef
parsing to recognize your for syntax? Your code will work in places
that multidefs can't right now (as Jakob noted) and we want to retain
that ability. I would be working on integrating for loops into TableGen
proper, not as a preprocessor.

+1 for not adding a preprocessor.

There seems to be two use cases for this:

1. Generating a sequence of similar top-level defs, for example registers named R0--R15.

2. Injecting patterns into multiclasses.

I could also imagine generating class fields:

class blah {
  for i = [1, 2, 3] {
    string Attribute#i#;
  }
}

Or even iterating base classes, which I've sometimes wanted to do:

class blah<list<SomeType> bases> :
  for c = bases {
    c<...>
  }
{
  ...
}

That seems much harder, though, primarily because TableGen currently
requires inheritance to be statically known. So I wouldn't go there
yet. :slight_smile:

It seems to me that the same mechanism could support both.

Yes, I agree. I'm going to play around with this and see if I can get
something working.

Che-Liang, I don't want to step on your toes, so let me know if I'm
disrupting something.

                                 -Dave

Jakob Stoklund Olesen <jolesen@apple.com> writes:

It's actually more complex than this but this gives the basic idea.
The key thing I need to be able to do is construct an instrinsic name
given some base string, a prefix and a suffix.

David, I think the syntax you are proposing here is much better than
the multidef syntax.

Yep, I agree.

When I look at test/TableGen/MultiPat.td, I have a really hard time
figuring out what is happening.

The multidef syntax is certainly esoteric and fixed by the for proposal.

I can see how the !foreach/!subst stuff could be daunting, but it is
really the key to writing completely generic patterns. Maybe I can
replace !foreach with for. I'll think about that. I think it would
clean things up a lot. !foreach has this odd iterator declaration mess
that for could potentially clean up. Plus it wouldn't have such a nest
of functional lisp-like stuff. :slight_smile:

I really only need !foreach to do stuff like what's in MultiPat.td so if
I can do that with for, I'd be thrilled to completely dump !foreach.

Here's a critical question. What type do the for loop iterators have?
string?

That's probably good enough.

One thing I do like about multidef is that the user gets to declare the
iterator and its type. Perhaps I'll try to do that with for.

                                -Dave

Jakob Stoklund Olesen <jolesen@apple.com> writes:

For example, I want to be able to do this:

defm MOVH :
vs1x_fps_binary_vv_node_rmonly<
   0x16, "movh", undef, 0,
          // rr
          [(undef)],
          // rm
          [(set DSTREGCLASS:$dst,
                (DSTTYPE (movlhps SRCREGCLASS:$src1,
                                (DSTTYPE (bitconvert
                                            (v2f64 (scalar_to_vector
                                                      (loadf64 addr:$src2))))))))],
          // rr Pat
          [],
          // rm Pat
          [[(DSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
            (MNEMONIC SRCREGCLASS:$src1, addr:$src2)],
           [(INTDSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
            (MNEMONIC SRCREGCLASS:$src1, addr:$src2)]]>;

This kind of thing is very hard to read and understand.

What's hard about it? I'm not trying to be agitational here. I'm truly
wondering what I can do to make this more understandable.

ISAs don't change a lot

Well, they do on some common architecutres. :slight_smile:

.td files are read lot more than they are written. I actually think
it is preferable to leave some redundancy in the .td files, it makes
it much easier to find things.

I had two goals when I designed our AVX implementation. Make it easily
extensible to longer vector lengths while keeping it as readable as
possible. I am certainly happy to make things more readable and welcome
lots of feedback in that area. But the ability to quickly and easily
extend the ISA for new vector lengths is critical to us. That drives
the generic pattern specification above and the !foreach/!subst stuff
you see in MultiPat.td. As I explained in another message, I think a
lot of MultiPat.td can be cleaned up with some semi-clever uses of the
for proposal.

I actually find the above easier to read and grasp that the multitude of
individual patterns for each different kind of ADD, SUB, etc. because I
don't have to go searching to find out how each one is defined. They're
all defined exactly the same way.

But of course, I've been working with this for a long time. I'm glad to
be getting some feedback on it now. :slight_smile:

                             -Dave

Jakob Stoklund Olesen <jolesen@apple.com> writes:

For example, I want to be able to do this:

defm MOVH :
vs1x_fps_binary_vv_node_rmonly<
  0x16, "movh", undef, 0,
         // rr
         [(undef)],
         // rm
         [(set DSTREGCLASS:$dst,
               (DSTTYPE (movlhps SRCREGCLASS:$src1,
                               (DSTTYPE (bitconvert
                                           (v2f64 (scalar_to_vector
                                                     (loadf64 addr:$src2))))))))],
         // rr Pat
         [],
         // rm Pat
         [[(DSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
           (MNEMONIC SRCREGCLASS:$src1, addr:$src2)],
          [(INTDSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
           (MNEMONIC SRCREGCLASS:$src1, addr:$src2)]]>;

This kind of thing is very hard to read and understand.

What's hard about it? I'm not trying to be agitational here. I'm truly
wondering what I can do to make this more understandable.

If you didn't write these patterns yourself, or if you wrote them six months ago, it is nearly impossible to figure out where a specific pattern came from, or where a specific instruction is defined.

It is hard enough mentally executing the current multiclasses. Injecting patterns into multi defms like this makes it much harder still.

I am certainly happy to make things more readable and welcome
lots of feedback in that area. But the ability to quickly and easily
extend the ISA for new vector lengths is critical to us.

This is where our priorities differ.

Readability and maintainability are key.

After all, we need to fix isel and codegen bugs more often than Intel and AMD add ISA extensions.

/jakob

Hi David,

In case you didn’t see my reply in llvm-commits. Let me repeat my objections here.

If I understand correctly, the reason you are pushing for all these extra abstraction features in TableGen language is to one day refactor SSE / AVX. The intention is to support future Intel ISA extensions which may extend AVX to vectors of arbitrary width. Am I understanding the purpose of these patches correctly?

Unfortunately I strongly disagree with the direction you are taking with this.

If we are to implement support for future Intel vector extension, we should add these instructions separately. I understand you are concerned about code duplication and would prefer to leverage what’s already written. However, our experience has shown the extra levels of abstraction is always high level that makes TableGen debugging harder than it should. Adding more layers, as you are proposing, would just make it unnecessary difficult. I believe it will be counterproductive.

I have discussed this carefully with Chris and have decided this is not the direction where we want to go with TableGen. We would really appreciate that you would back out these changes.

Thanks,

Evan

My purpose is to eliminate copy-paste style of programming in td files
as much as possible, but only to a point that the new language
constructs do not create too much overhead/readability-downgrade.

In other words, I am targeting those low-hanging fruit of copy-paste
programmings in td files that are eliminated by a simple for-loop
syntax. The repetitive patterns I observed in PTX backend (and
probably some other backends) are:
* Members of a multiclass, such as those in PTXInstrInfo.td.
* Consecutive similar defs, such as register declarations.

[Why for-loop?]
* All I need is a simple iteration language construct. I believe there
is no simpler constructs (in terms of readability) than a for-loop,
but I am happy to be convinced otherwise.
* It is sufficient to eliminate the most common copy-paste programming
that I observed.
* It is simple enough to understand and maintain, at least I believe so.

[Why preprocessor?]

I admit that a preprocessor is probably not the best solution. And we
can implement for-loop without a preprocessor. The only reason I chose
a preprocessor is because (I believe) this would add least changes to
the TGParser.cpp.

The TGParser.cpp as its current form parses and emits the results in
one-pass. That means it would emit the for-loop body even before we
are done parsing the entire for-loop.

So I believe a non-preprocessor approach would require 2 passes. The
first pass parses the input and generates a simple syntax tree, and
the second pass evaluate the syntax tree and emits output records (In
fact, this is how I implemented the current preprocessor). And I
believe that changing TGParser.cpp to accommodate 2 passes is quite a
lot, and so I chose a preprocessor.

But if you think we should really rewrite TGParser.cpp to parse and
evaluate for-loops correctly, I am glad that we could get away with a
preprocessor.

[Why NOT while-loop?]
* A while-loop requires evaluating an loop-condition expression; this
is complexity that I would like to avoid.

[Why NO if-else?]
* It requires at least evaluating a Boolean expression, too.
* If a piece of td codes is complicated enough that we need an if-else
to eliminate its duplication, I think it is worthy of the duplication.

[Why NO abstractions (like `define foo(a, b, c)`)?]
* Abstractions is probably worthy of, but I am not sure yet. I think
we could wait until it is clear that we really need abstractions.

Hi Dave and Jakob,

Thanks for comments. I try my best to respond any comments you wrote
about. If I missed any comments, as they are really a lot, please let
me know.

[string vs token]

The preprocessor (as its current form) has tokens by default, and it
only converts a series of tokens and white spaces into a string if
explicitly required (by a special escape #"#, see example below).

Jakob Stoklund Olesen <jolesen@apple.com> writes:

This kind of thing is very hard to read and understand.

What's hard about it? I'm not trying to be agitational here. I'm truly
wondering what I can do to make this more understandable.

If you didn't write these patterns yourself, or if you wrote them six
months ago, it is nearly impossible to figure out where a specific
pattern came from, or where a specific instruction is defined.

I haven't had that problem. Sure, I wrote the pattern but I'm really
not getting _what's_ hard about it. It looks just like any other
pattern with the types replaced by parameters.

In this example, all of the MOVH[PS][SD] patterns are defined right
there, in one place. IMHO this is much better than having the various
patterns scattered all over the .td file.

It is hard enough mentally executing the current multiclasses.
Injecting patterns into multi defms like this makes it much harder
still.

So this:

         // rm Pat
         [[(DSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
           (MNEMONIC SRCREGCLASS:$src1, addr:$src2)],
          [(INTDSTTYPE (movlhps SRCREGCLASS:$src1, (load addr:$src2))),
           (MNEMONIC SRCREGCLASS:$src1, addr:$src2)]]>;

is exactly the same patterns you see in some Pat<> defs today, except
those patterns are often completely separated from the "primary" pattern
defined by the defm (MOVH[PS][SD] in this case).

I find it easier to keep things in one place.

I am certainly happy to make things more readable and welcome lots
of feedback in that area. But the ability to quickly and easily
extend the ISA for new vector lengths is critical to us.

This is where our priorities differ.

Readability and maintainability are key.

Yes, they are very important.

After all, we need to fix isel and codegen bugs more often than Intel
and AMD add ISA extensions.

We need to do both. Less code generally implies easier maintenance.
I've maintained way too much cut-n-paste code in my career and it's
always a waste of time.

I am very open to reworking things to be more readable and maintainable.
I think that the for loop extension along with Bruno's enhacements to
allow multiclass defs to inherit from multiclasses will go a long way to
making that happen. These are both things that were on my TableGen
wishlist.

Give me a chance to try some things out and report back. Fair?

                                -Dave

Che-Liang Chiou <clchiou@gmail.com> writes:

My purpose is to eliminate copy-paste style of programming in td files
as much as possible, but only to a point that the new language
constructs do not create too much overhead/readability-downgrade.

Yes!

In other words, I am targeting those low-hanging fruit of copy-paste
programmings in td files that are eliminated by a simple for-loop
syntax. The repetitive patterns I observed in PTX backend (and
probably some other backends) are:
* Members of a multiclass, such as those in PTXInstrInfo.td.
* Consecutive similar defs, such as register declarations.

Yep.

[Why for-loop?]
* All I need is a simple iteration language construct. I believe there
is no simpler constructs (in terms of readability) than a for-loop,
but I am happy to be convinced otherwise.
* It is sufficient to eliminate the most common copy-paste programming
that I observed.
* It is simple enough to understand and maintain, at least I believe so.

I mostly agree with these. One other thing I've found useful is the
ability to abstract out the type information. For example, in x86 land,
an SSE/AVX add is always:

(set (type regclass:reg), (type (add (type regclass:reg), (type regclass:reg))))

Similarly, a sub is:

(set (type regclass:reg), (type (sub (type regclass:reg), (type regclass:reg))))

In fact most binary operations are:

(set (type regclass:reg), (type (op (type regclass:reg), (type regclass:reg))))

So why write hundreds of patterns to express this? Using the for-loop
syntax:

// WARNING: Pseudo-code, many details elided for presentation purposes.

multiclass binop<opcode> : sse_binop<opcode>, avx_binop<opcode>;

multiclass sse_binop<opcode> {
  for type = [f32, f64, v4f32, v2f64]
      regclass = [FP32, FP64, VR128, VR128]
      suffix = [ss, sd, ps, pd] {

    def !toupper(suffix)#rr : Instr<
      [(set (type regclass:$dst), (type (opcode (type regclass:$src1),
                                                (type regclass:$src2))))]>;
    def !toupper(suffix)#rm : Instr<
      [(set (type regclass:$dst), (type (opcode (type regclass:$src1),
                                                (type addr:$src2))))]>;
  }
}

multiclass avx_binop<opcode> {
  for type = [f32, f64, v4f32, v2f64, v8f32, v4f64]
      regclass = [FP32, FP64, VR128, VR128, VR256, VR256]
      prefix = [x, x, x, x, y, y]
      suffix = [ss, sd, ps, pd] {

    def V#prefix#NAME#!toupper(suffix)#rr : Instr<
      [(set (type regclass:$dst), (type (opcode (type regclass:$src1),
                                                (type regclass:$src2))))]>;
    def V#prefix#NAME#!toupper(suffix)#rm : Instr<
      [(set (type regclass:$dst), (type (opcode (type regclass:$src1),
                                                (type addr:$src2))))]>;
  }
}

def ADD : binop<add>;
def SUB : binop<add>;
def MUL : binop<add>;
def DIV : binop<add>;
[...]

Here I am treating "#" as an infix !strconcat. This makes things much
easier to read than both !strconcat() and a double-# notation, IMHO.

Now each binary pattern is only specified twice and even that
duplication can be eliminated with a little more abstraction. Perhaps
that's not worth it, however. I can live with this level of
duplication.

I would also like to replace #NAME# with a "real" Record field that is
automatically added to def-type Records. This removes a lot of the
hackiness of #NAME#.

[Why preprocessor?]

The TGParser.cpp as its current form parses and emits the results in
one-pass. That means it would emit the for-loop body even before we
are done parsing the entire for-loop.

It doesn't have to. I'm working on a for loop that is not preprocessor
based. It uses the same technique as the multiclass: remember the most
inner for loop seen and instantiate everything after the entire for loop
body is parsed.

So I believe a non-preprocessor approach would require 2 passes. The
first pass parses the input and generates a simple syntax tree, and
the second pass evaluate the syntax tree and emits output records (In
fact, this is how I implemented the current preprocessor). And I
believe that changing TGParser.cpp to accommodate 2 passes is quite a
lot, and so I chose a preprocessor.

In the long run, a two-pass parser would probably be more maintainable
but I agree it's a big job. That's why I went with a compromise of
treating for like a sort of multiclass, at least in the instantiation
mechanics. I'm still working on it so I haven't got all the details
together yet.

But if you think we should really rewrite TGParser.cpp to parse and
evaluate for-loops correctly, I am glad that we could get away with a
preprocessor.

I think we can do it without completely rewriting the parser.

[Why NOT while-loop?]
* A while-loop requires evaluating an loop-condition expression; this
is complexity that I would like to avoid.

I agree. It's not needed. Simple iteration over a list is enoguh.

[Why NO if-else?]
* It requires at least evaluating a Boolean expression, too.
* If a piece of td codes is complicated enough that we need an if-else
to eliminate its duplication, I think it is worthy of the duplication.

Perhaps. I'm sort of on the fence on this. I don't like the !if stuff
I introduced for readability reasons. An imperitive-style if might make
things easier but I think I could live with the duplication.

[Why NO abstractions (like `define foo(a, b, c)`)?]
* Abstractions is probably worthy of, but I am not sure yet. I think
we could wait until it is clear that we really need abstractions.

I'm not sure what you mean by this abstraction. Can you elaborate?

[string vs token]

The preprocessor (as its current form) has tokens by default, and it
only converts a series of tokens and white spaces into a string if
explicitly required (by a special escape #"#, see example below).

On further thought, I think this is correct. My plan with the
parser-integrated for is to allow the user to declare the type of the
iterator. It solves a number of problems, most importantly TableGen's
declaration before use requirement.

----------------------------------------
#for i = sequence(0, 127)
def P#i# : PTXReg<#"#p#i##"#>;
#end
----------------------------------------
* Anything between #"# are quoted, including white spaces and
non-tokens. E.g., #"#hello world#"# --> "hello world"

As mentioned above, I've been thinking about making "#" an infix
equivalent to !strconcat(). This would require the parser to implicitly
cast values to string if necessary but that's not hard to do.

* Macro variable needs a '#' character at both front and back. This
looks like the multiclass's #NAME# substitution, and so I think is
more consistent than prepending a single '#' at front.

To handle existing #NAME# constructs, I was planing to define a trailing
# as a !strconcat(string, ""). This keeps consistency and provides more
flexibility to the # operator.

What do you think? Which one is more readable to you? !case<> or #"# or ... ?

I like an infix #. It's consistent with at least some other languages.

[Can the for-loop proopsal be a preprocessing phase?]

I guess the example Dave gave (see below) cannot be handled in a (even
extended) preprocessor. I am not keen on implementing for-loop in a
preprocessor. I chose a preprocessor because I think it would cause
least impact to the codebase and, to be honest, I didn't address of
the pattern that Dave gave in his example in my design. I was trying
to avoid variable-length lists because I think that is too complicated
to users. But I could be wrong.

I think it could be if not done carefully. I am already getting
feedback that some of my proposals are too complicated. I am really
taking that to heart and trying to find a good balance among
maintainability, clarity and redundancy. I think things can be better
than they are now and better than what I've proposed so far. No one
gets the right answer in a vacuum. :slight_smile:

                              -Dave

David and Che-Liang, thank you for your hard work on this. Let me clarify what we may accept and what we cannot.

Che-Liang, we cannot accept TGPreprocessor. Please revert it as soon as possible. The for-loop syntax is still being discussed, it's not yet clear what we will end up accepting. Please work with Jakob on this.

David, we cannot accept the 'multidef' keyword. Please revert it. We find it very confusing and narrow is scope. We appreciate you thinking ahead about MIC, but we are against the massive refactoring and complicated abstraction scheme. We'll never accept those patches.

Your proposed new TableGen functionalities are interesting but it is clearly not where the code owners want it to go. Also, just so we are clear on this, Jakob is effectively the TableGen code owner. He will have the final say on all related issues.

Evan (putting on my code owner hat)

Evan Cheng <evan.cheng@apple.com> writes:

David, we cannot accept the 'multidef' keyword. Please revert it.

Working on it now.

We appreciate you thinking ahead about MIC, but we are against the
massive refactoring and complicated abstraction scheme. We'll never
accept those patches.

How about a less massive and complicated scheme? I think we can
make some good improvements to the current spec that will help
with the MIC work.

Your proposed new TableGen functionalities are interesting but it is
clearly not where the code owners want it to go.

Jakob at least seems interested in the for loop stuff. Am I reading you
correctly, Jakob? Having that feature would make a huge difference.

Frankly, the way things are with the x86 SIMD stuff is not scalable.
I'd like to clean some of that up, in an incremental way, of course.

                         -Dave

Evan Cheng <evan.cheng@apple.com> writes:

David, we cannot accept the 'multidef' keyword. Please revert it.

Working on it now.

We appreciate you thinking ahead about MIC, but we are against the
massive refactoring and complicated abstraction scheme. We'll never
accept those patches.

How about a less massive and complicated scheme? I think we can
make some good improvements to the current spec that will help
with the MIC work.

What do you have in mind? I don't think anyone is arguing that tblgen as-is is some paragon of virtue or anything. There's definitely room for improvements.