Iterating 'lit' RUN lines

As part of upstreaming the PS5 driver changes, I have a few cases of relatively large tests (as Driver tests sometimes are) where the behavior of PS4 and PS5 should be identical. For small tests I’ve just replicated the RUN lines and changed the triple, but is there a better way?

Ideally it would be nice to iterate some substitutions over a RUN line, so I could have
RUN: %clang -### -target %mytriple --foobar | FileCheck %s
and then just feed different triples into it.

Alternatively, if I can generate scripts with sed or whatever and then… recursively invoke lit to run them?

Any suggestions welcome.

The openmp/libomptarget test suite generates multiple sub test suites across all test files. Each generated sub test suite expands substitutions differently in order to target different devices.

However, sometimes it’s nice to iterate cases within a single test file (instead of across an entire test suite). For that, I have a lit extension in my downstream work that I’ve been meaning to upstream for years. It’s similar to the recent %if in that it preprocesses the RUN lines, but it’s a %for. It iterates data [edit: that is, substitutions] expressed in a %data directive. If you’re interested in this approach, I can explain more.

1 Like

I like the sound of the %for substitution: I can think of a number of test cases that do the same RUN line, but with different values for one small thing. Reducing duplication is always great, I think.

I found the openmp stuff; I see it generating a new lit.site.cfg per target, but it looks like it’s running the entire libomptarget test directory per target. That seems like overkill for a small number of tests.

So you have a %for feature, hmmm? Do tell!

I support the goal of finding ways to reduce duplication in lit tests, but I think we should try hard to keep things simple, and not let our pseudo-shell domain-specific commands-with-substitutions language grow too many features. I was already surprised by the complexity of %if.

Maybe this is the same as %for, but I can imagine a special kind of lit substitution where RUN lines containing the substitution in question are duplicated for all substituable values. Something like:
…substitutions.add(’%os’, MultiRunner([‘windows’, ‘linux’]))

I don’t think lit works that way… the user-friendly behavior would be to replicate the individual RUN line, but in practice lit pastes them all together into one moby command line (IIUC). Subsitutions are naive text replacement, with no notion of the larger “command” context. This is why %if ended up with the scoping that it has.

Here’s a simple example:

// RUN: %data opts {
// RUN:   (opt=-O0   )
// RUN:   (opt=-O1   )
// RUN:   (opt=-O2   )
// RUN:   (opt=-O3   )
// RUN:   (opt=-Ofast)
// RUN: }
// RUN: %data envs {
// RUN:   (env=           )
// RUN:   (env='env FOO=0')
// RUN:   (env='env FOO=1')
// RUN: }
// RUN: %for opts {
// RUN:   %clang %[opt] %s -o %t.exe
// RUN:   %for envs {
// RUN:     %[env] %t.exe > %t.out 2>&1
// RUN:     FileCheck -match-full-lines -input-file %t.out %s
// RUN:   }
// RUN: }

Of course, there are many details that can be debated about the
syntax, and I’m open to changing things. For example, I chose to use
the %[name] syntax to distinguish %for substitutions from other
substitutions. Also, given the way the %if syntax ended up, I
should probably use %{ and %} around the %data and %for
bodies.

Rather than trying to brain-dump all the details of how things
currently work, I’ll just answer questions as they arise.

While I find %for worthwhile in my work, I’ll admit up front that it
is inherently more complex than %if, both in interface and in
implementation.

I should also warn that my implementation needs a lot of cleanup. I
don’t expect the upstreaming process to be quick.

Making the lit meta-scripting language that much more complicated does seem like a heavyweight solution. I suppose another tactic would be to create some kind of preprocessing phase that lets a test define a parameterized macro, which it can then invoke multiple times. Just winging a syntax here:

PRERUN: define %option1(%triple) %clang -### --target=%triple --some-option %s | FileCheck %s --check-prefix=CHECK1
RUN: %option1(x86_64-scei-ps4)
RUN: %option1(x86_64-sie-ps5)
PRERUN: define %option2(%triple) %clang -### --target=%triple --other-option %s | FileCheck %s
RUN: %option2(x86_64-scei-ps4)
RUN: %option2(x86_64-sie-ps5)

and I suppose the macros could be brace-delimited allowing multiline definitions if that seemed worthwhile.

(Using a different prefix for the define directives so lit doesn’t have to interrupt itself while parsing the RUN lines; the macro expansion is not unlike regular substitutions. The main difference is that the individual test can define one, instead of that having to be in the lit.cfg file.)

Regardless of whether %for is worthwhile, I think it would be nice to be able to define substitutions directly within a single test file. I also think it would be nice to have function-like substitutions, whether defined in a single test file or in a config file so it can be shared among test files.

I like the ideas in this thread that keep lit as close as possible to being a simple template language. So, defining substitutions in a test seems like a nice way to stamp out RUN lines:

// DEFINE: %run=%clang -cc1 -fexceptions -std=N -mstuff %s
// RUN: %run -triple A | FileCheck %s --check-prefix=CHECK-A
// RUN: %run -triple A | FileCheck %s --check-prefix=CHECK-A

Per-test substitutions don’t reduce the number of RUN lines, which is what I was originally asking for. (Do these 50 RUN lines with --triple A, now run the same 50 lines with --triple B.) But it can allow refactoring some common things to shorten RUN lines, which certainly makes my particular need less tedious. Even without function-like substitutions, I could do something like

// DEFINE: %runps4=%clang -### --target=x86_64-scei-ps4
// DEFINE: %runps5=%clang -### --target=x86_64-sie-ps5
// RUN: %runps4 -fpic | FileCheck %s --check-prefix=FPIC
// RUN: %runps5 -fpic | FileCheck %s --check-prefix=FPIC

which makes it a whole lot easier to verify that everything is paired the way I want.

It does depend on permitting nested substitutions, which I’m not sure can happen today (I have a vague memory that there was a discussion of this some years ago and whatever the immediate problem was, it was solved without that). But even that fits in better with how lit tends to work today, rather than really complexifying the RUN line syntax.