AArch64 testsuite buildbots timeout


Some of our AArch64 bots have started timing out while compiling

On clang-cmake-aarch64-quick the failure first appears between r275337
and r275351, and on clang-cmake-aarch64-full it appears after r275352,
so there's isn't a clear culprit for this. I suspect we have been
slowly approaching that threshold for a while.

Both Renato and I are currently travelling, so we can't investigate
this until Monday. I can't really think of any temporary solution
other than bumping the timeout threshold, but IIUC that would affect
all the bots.

Sorry about the inconvenience,

We could split the file in half? Maybe a script based on int/float
would be roughly half and at least slightly principled.


That's another idea. Though, at the same time, Tramp3D-v4 is *also*
timing out, and only when building all tests on all cores, but not
when building it alone.

This could be a recent point change, but we're not tracking build time
individually. Though, if you see the total build time on:


Until build 275348, it was ~4h50, then jumped to ~5h10, which points
at some commit in that range that made compilation time much worse.

On the x86_64 board, it also seem to have jumped:


275310 was ~5h20 and 275356 was 5h55.

This could be affecting many tests, but tramp3d and unit tests more,
or just those, but it seems to be target independent.

The ARM builder hasn't got there yet, due to another bug adding a
build failure, but I expect to see the same problems...


An now, *all* self-hosting buildbots are timing out on stage 2:



I have a couple of powerpc bots that on occasion time out as well. It doesn't happen very often and I have been trying to tweak things to avoid it but it is still an ongoing issue.

At least a few of the factories have the ability to override the timeout setting on a per-slave basis. See the SanitizerBuildFactory and LLVMCMakeBuildFactory factories for instance. Perhaps it is time to add this to all (or at least more) of the factories.

It’d also be interesting to know why we’re slower on that range. I wouldn’t like to make it easier for people to tune the timeout, or we’ll end up regressing on compile time too much without noticing.


In the past I looked into the timeouts a bit and in many cases the time to run the particular test varied a lot from well under the specified limit to somewhere over it. When I ran them singly they never failed so I just attributed it to interactions between all the other stuff that was running slowing things down.