ARM failures

> Certainly slow builders are problematic. The phase-based building
> system David Dean is setting up may help mitigate some of this (it
> should make better use of the resources we have, as well as allowing
> us to benefit (in the form of smaller blame lists, though not
> necessarily lower buildbot result latency) from additional resources
> by allowing greater parallelism).

There's something I've always meant to ask: when you've got a stable buildot setup (same compiler, etc)
all the buildbots are still set up to do a configure and "make clean". I can easily understand if there was the possibility that
any other part of the system might also be being changed you'd want to be sure changes were due to repo changes
and so do a make clean. But those changes aren't too frequent and you could do manual clean at those times.

So all the make clean appears to be doing is guarding against an error in llvm's build system dependency checking. Is the tiny
probability of this worth the effect on the build times? (Personally I'd say no, but maybe there's an argument for it I haven't spotted.)

I don't believe so, no, and in fact when I went to setup my own
builders (for the GDB 7.5 test suite - I've got it running on the
public builder, but also my own internal one with a bit more hardware
so I can get results sooner (not ideal - I'd prefer to have things
publically, but I had the hardware lying around so figured I'd use
it)) I tried to do this. I seemed to have trouble with the 'configure'
step invalidating the whole build anyway - I don't want the configure
step to be something someone has to do manually on a new builder, but
I'm not sure how to run it in such a way that it doesn't cause an
otherwise incremental build to become a full rebuild either. Any ideas
would be most welcome.

Since most of lab's builders have been doing clean-rebuild, I let my
builders do incremental build.
In principle, the tree should be rebuilt (and reconfigured) from every
revision. I am addressing it on.
(That said, the real life is tough against the principle.)

I also suggest to introduce ccache, if reconfiguration could update
too many dependent files.

(I believe Takumi's bots do incremental rebuilds though, so I guess he
has some way of doing that)

With CMake builds, each builder watches
$(builddir)/{Makefile|ALL_BUILD.vcxproj|build.ninja}.
CMake runs only when they are missed.
I know a few issues around rebuild for reconfigration to fail.

With configure, each builder might watch $(builddir)/Makefile. It
works in most cases.
But it fails in a few cases. An extreme case is transition between
master and release.

FYI:
def CheckMakefile(factory, makefile="Makefile"):
    factory.addStep(SetProperty(name="Makefile_isready",
                                command=["sh", "-c",
                                         "test -e " + makefile + "&& echo OK"],
                                flunkOnFailure=False,
                                property="exists_Makefile"))

def Makefile_not_ready(step):
    return step.build.getProperty("exists_Makefile") != "OK"

factory.addStep(... doStepIf=Makefile_not_ready)