Recent buildbot failures on arxan_raphael and osu8

Hi,

I received a small flurry of buildbot emails yesterday. Most of them seem to be repeats of previous failures and buildbot only mailed them to me because the previous build raised an exception instead of failing. However I noticed a couple configuration/system issues amongst them:

· arxan_raphael is consistently hitting the 20 minute no-output timeout when running the link command in the compile step. It occasionally passes (e.g. build 11090) which suggests that it’s only slightly over.

· osu8 is out of disk space.

Daniel Sanders

Leading Software Design Engineer, MIPS Processor IP

Imagination Technologies Limited

www.imgtec.com

Daniel Sanders wrote:

Hi,

I received a small flurry of buildbot emails yesterday. Most of them
seem to be repeats of previous failures and buildbot only mailed them to
me because the previous build raised an exception instead of failing.
However I noticed a couple configuration/system issues amongst them:

·arxan_raphael is consistently hitting the 20 minute no-output timeout
when running the link command in the compile step. It occasionally
passes (e.g. build 11090) which suggests that it's only slightly over.

·osu8 is out of disk space.

Yup, I'm aware. osu8 runs everything in ram (all source and object files are in RAM), and the HD was never intended to take the load (small, slow, not redundant, and not expected to live a long life if it needs to spin that much). The Clang debug build -vg builder takes half the ram on its own at 17G, then it's running three other builders beyond that.

I had hoped that doing a make clean would fix its disproportionate footprint (the next largest builder is 5G), but apparently not so. I'm going to see whether we can buy another 32G RAM for it.

Sorry!

Nick

I suspect I’m hitting a RAM limit on that machine… either we need to increase the timeout or I need to move this task to a machine with more resources.

For the interim, let’s bump the timeout to 2400.

Is that something you can do Daniel?

Cheers,

Joe

I think this patch increases the timeout to 40 minutes for clang-x86_64-ubuntu only. It will need checking since I don’t have much experience with buildbot or the zorg scripts. Galina, could you take a look?

I also corrected an out of date comment. It seems the phased clang builders already use a 40 minute timeout but the comment still said 20.

arxan_raphael_40min.patch (2.24 KB)

Thanks for reminding me where these updates live.

I also need to indicate that I’m using gcc 4.8.2 now :slight_smile:

Joe

arxan_raphael_40min.patch (2.24 KB)