make check issue with llvm-cov

rkotler@mipsswbrd006-le:~/caviumllvm/build/test$ make
Making LLVM 'lit.site.cfg' file...
Making LLVM unittest 'lit.site.cfg' file...
( ulimit -t 600 ; ulimit -d 512000 ; ulimit -m 512000 ; ulimit -s 8192 ; \
       /usr/bin/python /home/rkotler/workspace/llvm/utils/lit/lit.py -s -v . )
XPASS: LLVM :: tools/llvm-cov/llvm-cov.test (8916 of 9784)
******************** TEST 'LLVM :: tools/llvm-cov/llvm-cov.test' FAILED

Is this a big-endian system? Yuchen had a tentative patch for that but it never went anywhere because he hadn’t figured out how to test it. If you’ve got a big-endian system and are willing to try out the patch, please let me know.

This board is little endian.

We do have a big endian machine too that I can try things on.

The immediate issue her for me is that llvm-cov is clearly failing but it returns 0.

This causes make check to fail because then it becomes an unexpected pass because
this test is xfailed for mips.

I don’t have a mips system, so it’s hard to know what’s going on. The XFAIL may be too broad. As I remember, the intention was to XFAIL big-endian systems. Then again, it does seem like there is a real failure here. I would be happy to work with you to try to get it fixed, but you’ll have to do some debugging to figure out what the problem is.

reed kotler <rkotler@mips.com> writes:

On a native mips linux machine, we are failing a make check test which
seems like it is supposed to already xfail.

I think you're mistaken. The test below looks like it's passing.

For one, the file test.c does not even exist (it is called test.cpp)

This should probably be test.o, rather than test.c or test.cpp, but the
interface to llvm-cov is compatible with gcov, which completely ignores
the file given as an argument other than to base the .gcno and .gcno
file names on it.

And in the test file there is a line:
XFAIL: powerpc64, s390x, mips, sparc

This is a crude attempt at "XFAIL: big-endian". The mips entry here is
just wrong if the system is little-endian - the test passes on
little-endian machines and fails on big-endian. This is obviously a
problem.

The script in make check thinks that it passed but clearly it does not.
llvm-cov is returning 0 in this case.

I'm not sure which case you're referring to. This test file really has
too many tests in it, which makes it confusing to debug failures.
Someone (me?) should probably split this into several tests that invoke
llvm-cov once each. The use of diff should be replaced with FileCheck as
well, so that error messages are clearer.

In any case, all of the llvm-cov calls except the read/checksum fail
invocations should (and do) return zero on little-endian. I think the
only problem here is that the XFAIL line is wrong.

> And in the test file there is a line:
> XFAIL: powerpc64, s390x, mips, sparc

This is a crude attempt at "XFAIL: big-endian". The mips entry here is just
wrong if the system is little-endian - the test passes on little-endian machines
and fails on big-endian. This is obviously a problem.

'XFAIL: mips' counts as an XFAIL for all mips targets because lit is checking for substrings (using the 'in' operator) of the target triple and 'mips' is a substring of 'mipsel-...'. I haven't tested this but I think that we need 'XFAIL: mips-, mips64-' to distinguish little/big-endian correctly for MIPS.
The relevant code is the isExpectedToFail() function in utils/lit/lit/Test.py.

That test cannot work at all right now because there is no test.c and that is being
called out for (there is test.cpp).

And in the test file there is a line:
XFAIL: powerpc64, s390x, mips, sparc

This is a crude attempt at "XFAIL: big-endian". The mips entry here is just
wrong if the system is little-endian - the test passes on little-endian machines
and fails on big-endian. This is obviously a problem.

'XFAIL: mips' counts as an XFAIL for all mips targets because lit is checking for substrings (using the 'in' operator) of the target triple and 'mips' is a substring of 'mipsel-...'. I haven't tested this but I think that we need 'XFAIL: mips-, mips64-' to distinguish little/big-endian correctly for MIPS.
The relevant code is the isExpectedToFail() function in utils/lit/lit/Test.py.

In this case what seems to be happening is the llvm-cov is NOT failing and it should.
That causes an unexpected pass which is what is making "make check" fail.

If you can help get it working on big-endian systems, we should be able to remove the XFAIL. That seems like the cleanest way out of this. Yuchen sent a patch to llvm-commits on 12/19/13. (I can resend it to you if you don’t have that.) Can you try that out on a BE mips system?

Even test.c does not exist.

Only test.cpp.

How can it pass?

I can try the patch over the weekend. I have to leave soon for today.

Even test.c does not exist.

Only test.cpp.

How can it pass?

See Justin Bogner's previous response to your query:

Sorry that I missed reading Justin's reply and for that reason did not understand Daniels point and Bobs point. My bad!

lol.

So there two things here:

1) I will try out the big endian patch today.

2) As Daniel pointed out, the Xfail needs to be more specific because Mips can be both little endian and big endian (we have both even on site at Mips/Imagination for testing).

3) If the patch works, then we can un Xfail all of the others hopefully.

Sorry for my poor reading of the discussion thread. I was distracted with other things I'm working on.

Reed

The patch cannot be applied as is anymore. Too much has changed.

I will study it a bit to see if I can simply fix it but I'm not committing myself to doing this.
I will see how much work that is.

In the meantime I propose fixing the XFAIL line as Daniel suggested. I will try that out in parallel to see if it fixes our other issues.

Reed

Reed