FNT testers reporting success even though they failed

All the LLVM lab FNT builders were reporting failures in the same set of tests
(scimark2, LivermooreLoops, some others). Now they've all turned green but the
same tests are still failing. I don't see any commits to deliberately make the
FNT pass, so it looks like someone broke failure detection... Note that the
FNT "report" text became empty at the same time. Here's what "report" was like
before:

http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12575/steps/make.test-suite/logs/report

and after:

http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576/steps/make.test-suite/logs/report

The first green builds are:
   http://lab.llvm.org:8011/builders/clang-x86_64-debian-fnt/builds/12576
   http://lab.llvm.org:8011/builders/clang-x86_64-linux-fnt/builds/57
   http://lab.llvm.org:8011/builders/dragonegg-x86_64-linux-gcc-4.6-fnt/builds/1983

Does anyone know what might have caused this?

Best wishes, Duncan.

[Thought it might've been my change (r169769) but that doesn't look
like it intersects with this particular test task]

I think you're on the right track with the logs. If the log that
reaches the buildmaster is empty then we won't find any failures &
we'll report success.

Have you checked any of the slaves to see if the log files are empty
in the builds themselves?

Have you run a local execution of the test suite uisng roughly the
same commands & observed whether the reports are empty coming out of
that?

- David

Hello everyone,

It might make sense to start from rebooting the buildmaster, just to make sure everything is all right on this end.
Yesterday I have tried to apply the latest changes from zorg and some of them are broken.
Theoretically, checkconfig shouldn't affect the working instance, but the reality could be different...
I planned to rollback to the last known-to-be-good revision and restart the master in the evening, but could do this sooner if you feel like this is urgent.

Please let me know.

Thanks

Galina

Hello everyone,

It might make sense to start from rebooting the buildmaster, just to make sure everything is all right on this end.
Yesterday I have tried to apply the latest changes from zorg and some of them are broken.
Theoretically, checkconfig shouldn't affect the working instance, but the reality could be different...
I planned to rollback to the last known-to-be-good revision and restart the master in the evening, but could do this sooner if you feel like this is urgent.

Given the state of things I don't think anyone will mind if you do it
sooner. Whenever's your soonest convenience sounds fine by me.

- David

Rebooted.

Thanks

Galina

The problem remains after the buildmaster restart.

Thanks

Galina

The problem remains after the buildmaster restart.

The FNT builders are now all failing again, and the "report" is no longer
empty. Does anyone know what fixed them?

Ciao, Duncan.

This issue is back: FNT bots are reporting success in spite of tests failing,
and the "report" text is empty again. Did someone change something?

Ciao, Duncan.

Hello,

I did reconfig the master on Thursday. No other changes on my side.

Thanks

Galina