RFC: Clang test runner changes

Hi all,

I thought I would give a little roadmap for where I am taking clang's
test runner. You can consider it an RFC and give feedback if you like!
:slight_smile:

Hi all,

I thought I would give a little roadmap for where I am taking clang's
test runner. You can consider it an RFC and give feedback if you like!
:slight_smile:

Sounds awesome to me!

-Chris

Daniel,

I’m so glad you’re taking on this work.

Any idea when it might be ready?

Just out of curiosity, what kind of program with the test runner be? C++ or a script of some sort?

It seems like a one-man job, but if you want any help, please let me know.

Meanwhile, I’ve got a VMWare virtual machine set up to do Linux, such that I’m thinking I’ll develop on Windows, but copy over the changes and run the tests on Linux, at least until your test runner is ready on Windows. I like the Visual Studio debugger too much…

Question: I just ran the tests on Ubuntu Linux, and it gave a message saying: “Expected Failures (17):” followed by a list of 17 files, presumably the tests that had failures. So basically this means that as long as the number of files listed matches the number given in the message, the overall test is considered successful, right? I know this is probably obvious, but I want to be sure. Perhaps a note about this in the “Testing” section on the hacking page would be a good idea.

-John

Just out of curiosity, what kind of program with the test runner be? C++ or
a script of some sort?

I'm assuming it'll stay in Python unless there's some reason why that
isn't sufficient.

Question: I just ran the tests on Ubuntu Linux, and it gave a message
saying: "Expected Failures (17):" followed by a list of 17 files, presumably
the tests that had failures. So basically this means that as long as the
number of files listed matches the number given in the message, the overall
test is considered successful, right? I know this is probably obvious, but
I want to be sure. Perhaps a note about this in the "Testing" section on
the hacking page would be a good idea.

You're okay as long as there isn't a section called "Failing Tests".
Perhaps the testrunner should be tweaked not to print expected
failures, though, since the information there can be easily gathered
using grep.

-Eli

In non-verbose mode, I don't want to see them either.

Removed, I agree it didn't make much sense. The original idea was we
didn't want to have XFAILs in the tree, but now that this is
semi-standard (at least till C++ is done) it is just noise.

- Daniel

Daniel,

I'm so glad you're taking on this work.

Any idea when it might be ready?

None; I would guess some substantive progress in more than a day and
less than a month. :slight_smile:

Just out of curiosity, what kind of program with the test runner be? C++ or
a script of some sort?

Python.

Question: I just ran the tests on Ubuntu Linux, and it gave a message
saying: "Expected Failures (17):" followed by a list of 17 files, presumably
the tests that had failures. So basically this means that as long as the
number of files listed matches the number given in the message, the overall
test is considered successful, right? I know this is probably obvious, but
I want to be sure. Perhaps a note about this in the "Testing" section on
the hacking page would be a good idea.

I removed the list, it wasn't useful any more.

- Daniel

Thanks for the info.

Question: I made a change and three tests are failing, but I can’t find any sort of log file or message that tells me what the errors were. Is there one in the test tree somewhere?

Of course I can manually run the compile, but it would be nice to have quick access to it after running the tests. A mention of it in the hacking Clang web page would be good too.

-John

John Thompson wrote:

Thanks for the info.

Question: I made a change and three tests are failing, but I can't
find any sort of log file or message that tells me what the errors
were. Is there one in the test tree somewhere?
Of course I can manually run the compile, but it would be nice to have
quick access to it after running the tests. A mention of it in the
hacking Clang web page would be good too.

All test results are in the Output subdirectory of clang/test. The
directory structure there should be the same as in the test source
directory. The interesting files are the .testresult files, since they
contain the command invocation and all output.

Sebastian

Running "VERBOSE=1 make" in the test directory should work.

Since you're figuring this all out at the moment, it would be great if
you could write it up all the confusing points. The website is in
SVN, so you can create patches to the website using "svn diff" and
submit them on cfe-commits.

-Eli

$ find test/Output | grep testresult
$

I think you need to update your tree!

But, to answer the question, the new method seems to be:

$ sh -x test/Output/temp.point/p1.cpp.script
+ llvm-build/Debug/bin/clang-cc -fsyntax-only -verify llvm/tools/clang/test/CXX/temp/temp.res/temp.dep.res/temp.point/p1.cpp
Errors seen but not expected:
   Line 19: non-const lvalue reference to type 'int' cannot be initialized with a value of type 'char'
Notes seen but not expected:
   Line 27: in instantiation of member function 'X0<struct N::B, int &>::test_f0' requested here

The testrunner doesn't create .testresult files anymore.

-Eli

Hmm, actually, there's something seriously screwy with the testrunner
at the moment: we don't actually support not and count, but tests
using them still manage to pass. I was just experimenting a bit with
not creating .script files, and somehow reading the commands from
stdin changes the relevant behavior. (The other issue with reading
the commands from stdin is that it appears to trigger some sort of
thread safety bug, but that isn't really relevant if you're planning
on rewriting the code.)

-Eli

You can get 'extra' output by default with:

- Implement the execution of RUN lines internally to the tool. The
format will probably become a bit more strict, instead of "sh" format.
The runner can then start to validate the lines, for example to ensure
that only certain tools are called (clang-cc, count, not, FileCheck,
etc.). The runner can then also implement certain features internally
(like grep).

Hmm, actually, there's something seriously screwy with the testrunner
at the moment: we don't actually support not and count, but tests
using them still manage to pass.

I'm not sure what you mean by "we don't actually support not and
count". The test runner modifies the PATH of the subprocess to include
LLVM's test/Scripts dir, which is the same as it used to do (although
the mechanism changed a bit).

I was just experimenting a bit with
not creating .script files, and somehow reading the commands from
stdin changes the relevant behavior. (The other issue with reading
the commands from stdin is that it appears to trigger some sort of
thread safety bug, but that isn't really relevant if you're planning
on rewriting the code.)

I'm not exactly sure what problem you are referring to, but yeah
perhaps not worth worrying about until things change to not using
/bin/sh.

- Daniel

Oh, oops, my mistake; I didn't realize that the Makefile does that...

-Eli

Okay... if you're interested anyway, I've attached the patch.

-Eli

x.txt (1.22 KB)

Daniel,

I'm so glad you're taking on this work.

Any idea when it might be ready?
Just out of curiosity, what kind of program with the test runner be? C++ or
a script of some sort?

It seems like a one-man job, but if you want any help, please let me know.

FYI, this has progressed far enough that -- with local patches, and
some setup -- it is possible to get interesting test results on
Windows.

Currently on my machine with an assortment of patches I'm down to 121
failures, almost all of which are failures not related to the testing
infrastructure. If you are interested in getting things going further,
a good place would be to start looking at those failures and how to
fix them. If you want to take this one, the first step is to get the
test runner working in your setup from the clang-test project (this is
probably only a matter of passing --no-sh in the custom build step,
and maybe tweaking some paths). If that works for you I can pass along
my extra patches to cut down on the spurious failures (adding
count.bat and not.bat in llvm/test/Scripts is the most major one).

- Daniel

Would it be possible for the .script file to include the PATH change in its text so it's easier to run the .script file manually to reproduce failures?

Shantonu Sen
ssen@apple.com

I got hit by this just today... I'd like it as well. :slight_smile: