Buildbots do not detect test-suite failures!!!

It has just been brought to my attention that our buildbots no longer report failures to compile/link files in the test-suite as failures for the build. As a result, we have at least one bot that is currently showing a green status, yet it aborts when building 3 test cases.

I assume that something has changed with how (presumably LNT) reports these failures and the Zorg code does not adequately detect this (I think the buildbot scripts grep for a string such as “FAIL:” in the log or something along those lines). Previously, a failure to compile would show up as two failures (a failure at compile time and a failure at link time). Now it just shows up as NOEXE.

This really seems like something we should fix ASAP.

For an example:
http://lab.llvm.org:8011/builders/clang-s390x-linux-lnt/builds/11252/steps/test-suite/logs/stdio

The Vector-build test case shows up as NOEXE and we’re clearly hitting an UNREACHABLE while compiling the test:

Unexpected node to lower
UNREACHABLE executed at ...

I have sent this a bit over a month ago and got no response. Does anyone know if this issue has been resolved? I hope that we are no longer in a situation where we may have LNT failures on build bots that aren’t reported as failures.

Speaking of LNT, we've had a lot of trouble setting it up locally. We
can get it to run basic pass/fail tests all right, but any performance
tracking is pretty hopeless. From talking with others, their
experiences seem to have been the same.

It would be nice to have a non-bespoke system to test larger
applications and track performance for them. Has anyone looked into
options? Would such an effort be welcome, or are people happy with LNT?

                        -David

Nemanja Ivanovic via llvm-dev <llvm-dev@lists.llvm.org> writes: