Please help: SVN is broken

Hi all, for the past several days I’ve been dealing with intermittent bot failures due to a recent series of patches which changes some things about lit.

Most of the dust has settled, but there’s several bots that just won’t stay green.

An example is this one:

http://lab.llvm.org:8011/builders/llvm-hexagon-elf

What happens in all of these cases is that SVN encounters a tree conflict when updating, and then it gets stuck on some messed up of the code where the test suite doesn’t match up with the code being tested.

But at this point I honestly think this is some kind of bug in SVN or buildbot and I’m not entirely sure what to do about it. For these affected bots, I can force a clobber by setting build revision to 0 and forcing a build. This will rm -rf the source tree and check out a new one. Surely this should resolve any conflicts right? But it doesn’t. For example:

In this build I clobber the source tree. You can see by clicking the svn-llvm step. You can see that all tests pass.
http://lab.llvm.org:8011/builders/llvm-hexagon-elf/builds/9746

The next build, 9747 also passes and if you click the svn-llvm step you’ll see that it just updates some files.

But all of a sudden, 9749 fails. If you look at the svn-llvm step again, you see this:

Updating ‘.’:
Skipped ‘utils/lit/lit/llvm’ – Node remains in conflict

and now everything starts failing.

But it shouldn’t even be trying to update this! No file in that directory has changed. If you look at 9747, 9748, and 9749, none of those CLs touch any files in that folder.

So, I’m really at a loss here. I don’t think reverting is going to fix anything, because I believe the damage to be related to some intermediate commit, and the revert is just going to get stacked onto the end of the commit list. Something seems fundamentally messed up here.

Can anyone provide some insight?

There are two buildslaves hooked up to that builder, and those two builds did not run on the same slave.

-Eli

On llvm-hexagon-elf, the two builds you mention actually came from two different machines. One of them fails consistently, the other one works. Since the builds are dispatched to whichever one is free, it may appear that they fail randomly. Let me see if I can fix it, if not, I'll complain to someone who can.

-Krzysztof

It seems like everything is getting back to normal? (Disregarding the fact that every bot is currently red, but at least it’s an actual bug in someone’s patch this time)

Thanks