Porting F18 tests to use LLVM lit

Hi,

Further to the conversation in llvm-dev, regarding LLVM testing infrastructure/lit, and the successfully merged patch from Carol, we’re continuing development on adding this infrastructure by porting the existing tests to be compatible with lit.

We’d like to propose the following approach:

Thanks for working on this!

Further to the conversation in llvm-dev<http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html>, regarding LLVM testing infrastructure/lit, and the successfully merged patch from Carol<https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245>, we’re continuing development on adding this infrastructure by porting the existing tests to be compatible with lit.

We’d like to propose the following approach:

  1. Start off with a new directory, “test-lit” containing lit tests
  2. make check-all will run this and the old test directory together
  3. port over all old tests to this new directory as lit tests
  4. Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.
  5. All new tests will be created to be compatible with lit.
During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.

This sounds very reasonable.

My PR has three patches:

  1. A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh
  2. The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script
  3. All the remaining error tests from semantics ported to lit.

Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We
might want to add a scripts there if we can generate, not port, the
CHECK lines.

Cheers,
  Johannes

Hi Luke,

During the transition, when writing a lit test, are there any special guidelines beyond choosing a distinct test name?

After the final step of removing old tests, will you rename the test-lit directory to test?

Thanks for working on this!

- Steve

    External email: Use caution opening links or attachments
    
    Thanks for working on this!
    
    > Further to the conversation in llvm-dev<http://lists.llvm.org/pipermail/llvm-dev/2020-January/138282.html>, regarding LLVM testing infrastructure/lit, and the successfully merged patch from Carol<https://github.com/flang-compiler/f18/pull/861/commits/1049aeb81e0195d3fa1050b21e93a3980a0b7245>, we’re continuing development on adding this infrastructure by porting the existing tests to be compatible with lit.
    >
    > We’d like to propose the following approach:
    >
    > 1. Start off with a new directory, “test-lit” containing lit tests
    > 2. make check-all will run this and the old test directory together
    > 3. port over all old tests to this new directory as lit tests
    > 4. Once all tests are successfully ported, the old directory and test scripts are removed, and “test-lit” becomes “test”.
    > 5. All new tests will be created to be compatible with lit.
    > During this transition phase, tests can either be written in the old style, and then ported with my script, or preferably, written to work with lit.
    
    This sounds very reasonable.
    
    > My PR has three patches:
    >
    > 1. A script which ports old style tests to new lit tests. At the moment it only handles semantic error tests, i.e. tests run with check_error.sh
    > 2. The output of the porting script on a subset of the tests for verification. The subset of tests should cover all variations of error tests in my script
    > 3. All the remaining error tests from semantics ported to lit.
    
    Did you look at the llvm/utils/update_XXX_test_checks.py scripts? We
    might want to add a scripts there if we can generate, not port, the
    CHECK lines.
    
    Cheers,
      Johannes

Hi Stephen,

I don’t see any issue with choosing test names, I can confer if you like but the naming of the existing tests have never come into question before, so I doubt anyone would have any particular preferences. They’re in a separate folder, so make sure the test name isn’t used in the existing tests, then I can’t imagine any problems arising.

And yes, when all tests are successfully ported, ‘test’ will be removed, and ‘test-lit’ will be renamed to ‘test’ taking its place, and the appropriate changes needed to incorporate this into make check-all will be needed as well, but should just be as simple as renaming again.

Both swift [1] and clang [2] have built-in capabilities to validate semantic checks. These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.

Here’s a sample line from a clang test [2]:

template<> struct __is_floating_point_helper<__float128> {}; // expected-error {{__float128 is not supported on this target}}

From swift [1]:

private func publicReq() {} // expected-error {{method ‘publicReq()’ must be declared public because it matches a requirement in public protocol ‘PublicProto’}} {{none}}

Both clang and swift use lit.

Here’s the lit line from the clang test [2]:

// RUN: %clang_cc1 -verify -std=c++11 %s

From swift [1]:

// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5

FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]

The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit. An alternative form of this proposal is to duplicate the f18 test scripts in python for portability.

  • Steve

[1] https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift

[2] https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp

[3] https://github.com/flang-compiler/f18/pull/934

Both swift [1] and clang [2] have built-in capabilities to validate semantic checks. These compilers, like f18 and GCC, use a special markup that is designed to verify diagnostics and to test semantic analysis.

Here’s a sample line from a clang test [2]:
  template<> struct __is_floating_point_helper<__float128> {}; // expected-error {{__float128 is not supported on this target}}

From swift [1]:
  private func publicReq() {} // expected-error {{method 'publicReq()' must be declared public because it matches a requirement in public protocol 'PublicProto'}} {{none}}

Both clang and swift use lit.

Here’s the lit line from the clang test [2]:
// RUN: %clang_cc1 -verify -std=c++11 %s

From swift [1]:
// RUN: %target-typecheck-verify-swift -enable-objc-interop -disable-objc-attr-requires-foundation-module -swift-version 5

FileCheck appears to be a very powerful tool; however, using FileCheck markup to verify diagnostics is not as straightforward as the markup used in swift and clang tests. [3]

The f18 semantic tests and the f18 tools used for verifying diagnostics are similar in spirit to clang’s and swift’s. We could adapt the existing f18 test scripts, e.g. test_errors.sh, to work with lit.

I believe that we should use FileCheck where possible.

-Hal

An alternative form of this proposal is to duplicate the f18 test scripts in python for portability.

- Steve

[1] https://github.com/apple/swift/blob/master/test/Sema/accessibility.swift
[2] https://github.com/llvm/llvm-project/blob/master/clang/test/Sema/128bitfloat.cpp
[3] https://github.com/flang-compiler/f18/pull/934

MLIR is using the same principle: mlir-opt supports a --verify-diagnostic flag to check for diagnostics.
Here is an example test:

https://github.com/llvm/llvm-project/blob/master/mlir/test/AffineOps/invalid.mlir

This change should proceed in two steps:

  1. Run tests with llvm-lit instead of ctest but don’t change how tests are checked.

  2. Replace custom checking of results with FileCheck where appropriate.

The first step should be relatively straight-forward. Once this transition is complete the rest can happen incrementally.

The second step is more complicated because there are several different kinds of tests with their own requirements for checking the results. The folding tests might be simplified by using FileCheck. I don’t know how .mod file tests and symbol tests can use FileCheck (but my knowledge of FileCheck is limited). There is on-going discussion in PR #934 about how to use FileCheck on the error tests.

Obviously the overriding consideration is making it easy to write high quality tests, run them, and diagnose their failures. Anything we can do to improve that will pay off greatly in the future.

Tim

Hi Tim

I really like this idea because it allows us to tackle one of the obstacles towards LLVM unification that is non-controversial on our side and then give proper thought to the more controversial choice of how/if to migrate off the existing checking scripts in F18. Luke’s going to change direction a bit to explore this avenue.

He's still working on showing off what FileCheck can do to replace test_errors to further the discussion that started on PR#934. As the above approach would cause those patches to change a lot, Luke will start a new PR for the alternative approach you propose and add all of the reviewers of PR#934 over as reviewers.

Ta
Rich

I somehow got Rich’s email twice, so I’ll reply to the first one.

As promised, I’ve submitted a new pull request here that adds lit, without FileCheck. This PR just looks at error tests, and I imagine people will want the remaining test types (semantics[symbol, modfile, generic], preprocessing, decimal, evaluate and runtime) to be ported using the original shell scripts, before swapping them out for FileCheck (which I will be working on simultaneously).