[lit] Run a RUN line multiple times with different % replacements

[MaskRay]

@jdenny-ornl

I acknowledge that multi-valued substitutions like %{stdcxx98-14} might be tailor-made for the use cases originally presented in this discussion. In comparison to the approach I’m suggesting, that might make them initially easier to use once nuances are well defined and understood. But will multi-valued substitutions still be useful as those use cases become more complex? Is a user’s understanding of those nuances reusable for other use cases? Is the implementation in lit reusable for other use cases? For the approach I’m suggesting using per-test substitutions (via DEFINE/REDEFINE), my hope is that the answer is yes to all these questions. But of course, there will be nuances there too.

I agree that %{check-stdcxx_17-} making the containing command repeated is magical. I pick it mostly because it requies the minimum amount of efforts to implement…
I think using a generic design like your ⚙ D132513 [lit] Implement DEFINE and REDEFINE directives is better. I just want to understand (a) how difficult it is to support function-like macros and
(b) how long ⚙ D131464 [test] Make tests pass regardless of gnu++14/gnu++17 default should wait on ⚙ D132513 [lit] Implement DEFINE and REDEFINE directives and its follow-ups.

For D131464, If I understand correctly, we can change clang/test/lit.cfg.py this way:

config.substitutions += [
  ('%{check-std:cxx98-14}', '%{check-std}(c++98) && %{check-std}(c++11) && %{check-std}(c++14)'),
  ('%{check-std:cxx17-}', '%{check-std}(c++17) && %{check-std}(c++20) && %{check-std}(c++2b)'),
]

In a clang/test/ test, write

// REDEFINE: %{check-std}(STD) = %clang_cc1 -fsyntax-only -verify=expected,precxx17 -triple=x86_64-linux-gnu -std=%{STD} %s
// RUN: %{check-std:cxx98-14}

// REDEFINE: %{check-std}(STD) = %clang_cc1 -fsyntax-only -verify=expected,cxx17 -triple=x86_64-linux-gnu -std=%{STD} %s
// RUN: %{check-std:cxx17-}

I have noticed that your example uses // DEFINE: %{check-std}( STD %, PREFIXES %) =. Some questions and thoughts:

• %, appears to be a argument separator. Is this deliberately preferred over ,?
• The parameter list end uses %) instead of ). Is this better?
• I try to avoid too many punctuations in %{...}, so I pick %{check-std:cxx17-} instead of %{check-std:c++17-} . Do I need to avoid +?
• For now we may assume that some metacharacters such as (, ), , cannot appear in macro arguments.

lit - LLVM Integrated Tester — LLVM 18.0.0git documentation has a few %{...} uses. There aren’t many predefined substitutions, so I think overloading it with macro arguments is fine.