Self-hosting bots noise

Hi David/Galina,

FYI, I found one big reason why self-hosting bots fail long after the
offending commit is tested: dirty builds.

In a dirty build (no make clean), when a file X is changed that breaks
file Y, the bot will only accuse the failure when file Y is changed,
too, so that stage1 with modified X will compile file Y, and break.

At hindsight, it's obvious. But I hadn't connected the dots before... :confused:

I'm moving all my self-hosting bots to clean builds and will be
relying solely on ccache to make stage1 fast. This shall help with
"breakages not on the blame list" problems.

cheers,
--renato

Hi David/Galina,

FYI, I found one big reason why self-hosting bots fail long after the
offending commit is tested: dirty builds.

In a dirty build (no make clean), when a file X is changed that breaks
file Y, the bot will only accuse the failure when file Y is changed,
too, so that stage1 with modified X will compile file Y, and break.

That sounds like a bug in the build system - do you have any examples of
this? We should fix them.

The build should be reliable - any case where a clean build produces a
different result from a dirty build reduces/removes confidence in
incremental builds & hurts developer productivity (just like flakey tests,
etc).

- David

I think he meant that the /second-stage/ build should always be done as a clean build – to make sure that all of the source gets rebuilt with the newly-compiled compiler, instead of only rebuilding the modified sources with the newly-built compiler.

Build systems typically aren’t expected to watch out for the compiler itself changing out from under them.

+1, I know that my self-hosting Windows buildbot does not do a clean second
stage build, and I've been meaning to fix that for some time.

Renato, I would rather fix this in our buildbot configs. We should just
make stage2 always be clean, and that will fix my builders as well as yours.

Yeah, sorry, a stage2 is always necessarily clean - because there's a new
compiler. ccache isn't even relevant/useful here, is it (I mean it should
be a no-op, if it hashes the compiler itself and all the compiler's
dependencies too - but I don't know if it does that?)? We want to actually
do the real work of compiling the code with the new compiler for
everything. Nothing should be a cache/build hit. (fancy build system like
Bazel could still get a cache hit on the link steps - if the compiler
produced identical object files, etc)