Test case paths in llvm-lit output are lacking the location of the test dir itself

Beginners
1

I’ve been using llvm for years and typically when running lit tests my current directory is llvm/.

Then I often run a subset of tests, for example by running commands such as
build/bin/llvm-lit test/CodeGen
or
build/bin/llvm-lit ../clang/test/Analysis
etc.

In case of test failures I get output such as

PASS: LLVM :: CodeGen/X86/vec_int_to_fp.ll (1525 of 1525)
*****************
Failed Tests (2):
  LLVM :: CodeGen/MyTarget/foo.ll
  LLVM :: CodeGen/SomeTarget/bar.ll

Then I usually want to do something related to the failed test case (e.g. rerun llvm-lit on those, or edit them, or run script to update the test checks).

The thing that has annoyed me for years is that the paths written isn’t relative to my current dir. So I can’t just copy-n-paste the paths to the test cases, because I always need to manually prepend “test/” or “../clang/test/” etc. when copying those paths.

Anyone who knows about llvm-lit and the testing framework who knows if it could be possible to let llvm-lit print paths relative the to current dir instead (or absolute paths)? Maybe there is some option already that can be used?

3 Likes
2

This doesn’t quite answer your question, but if you run with -v (for failing only) or -a (for all), the tests print the commands that are run as absolute paths, although it doesn’t include the actual working directory. Depending on the test, it may be obvious from the output what that directory is, since it’ll often include the expansion of %t in a file output from one of the commands.

I’d happily accept a lit enhancement that prints the test output directory somewhere in the verbose log though.

3

Well, the most common use case is that I want to rerun the failing tests using -av option to get those full command lines for the failing tests (but only the failing tests without getting thousands of lines with passed tests).

4

So maybe -sv is a good option.

But in situations when I just want a summary of failed tests to quickly copy-n-paste those paths in to a cmd line the open them in an editor, or feeding them into update_test_check.py, then that wouldn’t help much.

5

I’ll just add my +1 that the current behavior is very annoying. Being able to copy & paste test names is such an important workflow. Adding that llvm/test/ before the name is not exactly hard, but you have to do it all the time.

A possible complication with changing this is options like LIT_FILTER out, that work on the LLVM :: CodeGen/MyTarget/foo.ll format. I guess we can just keep those working against the test suite root, even if we print the failures against the current directory, though that does create something of a mismatch…

6

I’ve been annoyed by this for years. I looked into this the other day, and came to the conclusion that there wasn’t a flag for it, but that it would be quite easy to adjust the thing that prints these to drop the ${suite} :: prefix, and insert the path relative to $(pwd) instead.

There is –-order=smart, but it appears untested. It would be nice to have –-order=failing or somesuch to re-run only failing tests.

7

After writing this I looked a bit into llvm-lit myself.

A small one line change like this seem to result in full paths:

diff --git a/llvm/utils/lit/lit/Test.py b/llvm/utils/lit/lit/Test.py
index 729097772417..3a8e615c9d97 100644
--- a/llvm/utils/lit/lit/Test.py
+++ b/llvm/utils/lit/lit/Test.py
@@ -305,7 +305,7 @@ class Test:
         return self.result.code.isFailure
 
     def getFullName(self):
-        return self.suite.config.name + " :: " + "/".join(self.path_in_suite)
+        return self.suite.config.name + " :: " + self.getFilePath()
 
     def getFilePath(self):
         if self.file_path:

I’ve not tested it much, and only on linux. But if that at least could be enabled using some llvm-lit option, then I think it could be useful to me.

8

Made an attempt to add an option to llvm-lit to print paths relative to current working dir:
[lit] Add support to print relative paths when outputting test names by bjope · Pull Request #154317 · llvm/llvm-project · GitHub