Buildbots do not detect test-suite failures!!!

It has just been brought to my attention that our buildbots no longer report failures to compile/link files in the test-suite as failures for the build. As a result, we have at least one bot that is currently showing a green status, yet it aborts when building 3 test cases.

I assume that something has changed with how (presumably LNT) reports these failures and the Zorg code does not adequately detect this (I think the buildbot scripts grep for a string such as “FAIL:” in the log or something along those lines). Previously, a failure to compile would show up as two failures (a failure at compile time and a failure at link time). Now it just shows up as NOEXE.

This really seems like something we should fix ASAP.

For an example:

The Vector-build test case shows up as NOEXE and we’re clearly hitting an UNREACHABLE while compiling the test:

Unexpected node to lower
UNREACHABLE executed at ...

I have sent this a bit over a month ago and got no response. Does anyone know if this issue has been resolved? I hope that we are no longer in a situation where we may have LNT failures on build bots that aren’t reported as failures.

Speaking of LNT, we've had a lot of trouble setting it up locally. We
can get it to run basic pass/fail tests all right, but any performance
tracking is pretty hopeless. From talking with others, their
experiences seem to have been the same.

It would be nice to have a non-bespoke system to test larger
applications and track performance for them. Has anyone looked into
options? Would such an effort be welcome, or are people happy with LNT?


Nemanja Ivanovic via llvm-dev <> writes: