Many bots don't build anything -- does anyone know why?

Project Infrastructure LLVM Dev List Archives
1

Hi,

many of the bots on http://lab.llvm.org:8011/console don’t do anything in their compile phase, even if they should.

For example, these bots all don’t do anything in their compile phase in any builds, even if they should:
http://lab.llvm.org:8011/builders/clang-x64-ninja-win7/

http://lab.llvm.org:8011/builders/clang-ppc64be-linux

http://lab.llvm.org:8011/builders/clang-atom-d525-fedora-rel

http://lab.llvm.org:8011/builders/clang-cmake-thumbv7-a15

http://lab.llvm.org:8011/builders/clang-s390x-linux

At least the ninja-win7 bot used to work fine last week Thu and early Fri, so this looks like a recent regression. The clang_fast bots seem to still work, so we’re not flying completely blind, but we for example don’t have Windows coverage at the moment. Does anyone know what changed? How can we get working bots again?

Thanks,
Nico

2

Hi Nico,

Thanks for reporting this. I restarted clang-cmake-thumbv7-a15 and cleared its ccache, I’ll be keeping an eye on it.

Regards,
Diana

3

Hi,

It seems the problem is that the bot is updating llvm.src, but trying
to build from llvm.

Galina, it looks like this is related to your recent changes to zorg,
you probably missed a spot somewhere and I can't find it on a first
glance. Could you have a look?

Thanks,
Diana

4

Also, I have removed the llvm directory from the bot and moved it to
the silent master until we fix this.

Diana

5

Right, this affected all of our bots, as files became stale when a
test was moved.

Now, it seems, all the buildbots are green, not because they're
passing, but because they're building from a time when they were
passing, and the SVN update is making no difference... :S

All our dirty bots are taking 1s to build (no changes) and our clean
bots are taking 8 minutes (ccache's work).

Should we revert this? This is pretty serious? Can we fix this quickly
on the bots?

--renato

6

Just FYI, I'm moving the "llvm" directory away and making it as a
symlink to "llvm.src", at least until we sort out this problem.

Since this has been going on from Sep 1st, I believe we'll find *many*
breakages in the bots... :frowning:

cheers,
--renato

7

That's 4 days, and the clang_fast bots are still testing things. I expect
that some stuff will be broken, but I don't think it'll be _that_ bad.

+1 to fixing quickly, but from what I understand just reverting isn't
enough since the master needs to be restarted after the revert for it to
have an effect.

8

That's 4 days, and the clang_fast bots are still testing things. I expect
that some stuff will be broken, but I don't think it'll be _that_ bad.

Ahn, that's good to hear.

From our side, two of the bots have returned, and both are broken on

many (seemingly unrelated) tests. And they were the simplest ones.

Bisection will be hard with breakages on top of others, and it will
take a while to clean it up, which means more failures will trample
over...

Last time this happened to one of our bots, it took me three weeks to
go back to green. If even only half of our bots go red in that state,
we may be in it for the long run... :frowning:

+1 to fixing quickly, but from what I understand just reverting isn't enough
since the master needs to be restarted after the revert for it to have an
effect.

Indeed. It has to be Galina.

cheers,
--renato