Wanted: a way to test changes before breaking all the build bots.

So, I broke a few build modes with a recent commit (r266002). I had been developing with clang in debug mode, which worked and passed tests fine. But upon committing, I discovered that I had broken:

  1. Bots building with GCC. (fixed subsequently in r266011)
  2. Bots doing an NDEBUG build with -Werror. (fixed subsequently in r266016)
  3. Bots building llvm with msan. (fixed via revert r266062)

Obviously, that was not an ideal outcome.

I’d love to have some reasonable way that I could try out a commit other than submitting it and hoping for the best, so that sort of thing can be avoided.

Ideally, I could run a change through the actual buildbots and get back a report. I’m not sure if that’s possible or not. Or, if it would overload the build machines, since many of them seem barely even able keep up with the submit rate. (it would still be valuable even if the feature was restricted to platforms easily obtainable at arbitrary scale on public clouds, though.)

But, barring that, even just having a script that would automatically run all of the builds on my own machine, equivalent to what the bots running the same host platform would run, would be a great help. With that, I could at least be fairly sure my commit wouldn’t break any Linux/x86-64 bots, which isn’t really reasonably possible right now.

Is anyone working on anything to make this situation better?

Hi James,

The proposal for pre-commit buildbots is a long lived one...

The accepted reality (not a consensus), is that it's more
cost-effective to spend time and money on real builders for new
configurations than duplicating builders for pre-commit. Given that we
have a lax policy on reverting patches anyway, people feel comfortable
in breaking a few things here and there.

Personally, I would love some pre-commit buildbot. I really dislike
the series of fix-up patches that come after breaking bots, making
them impossible to revert.

I'm assuming that the number of pre-commit patches are larger than
commit ones, given that people will be trying them out as they bake.
Therefore, such a pre-commit infrastructure would have to be bigger,
per configuration, than our current buildbot infrastructure. If we
have one builder per architecture, but multiple slaves, that could be
scalable, but they wouldn't catch all bugs (which is ok).

We'd also want all those pre-commit builders to be monitoring the same
branch across all projects. And that's where testing patches that
cross project boundaries become complicated. LLVM has around 12 major
repositories, that get built and tested regularly, and the algorithm
that would pick a branch and test in Buildbot is not trivial.

The alternative would be Jenkins, uploading a patch (that could
potentially span across multiple repositories), and then test. The
infrastructure would be *much* simpler, but we'd have to create a
parallel infrastructure, not necessarily synchronised with the main
one. Given that Apple is moving to Jenkins internally, I don't see
this as a bad thing, at least not for a completely separated
infrastructure.

But both solutions will have the volume issue. You may have to wait
longer to have your patch validated on a tree that might be, now,
outdated, than to commit, break & fix. This is a somewhat contentious
issue in this community.

My opinion is that we should have less haste, more quality. So,
pre-commit bots (BB or Jk or whatever) is a way forward, and I support
it. But I have been frequently outnumbered in the past... :slight_smile:

cheers,
--renato

I don't think we need perfect/complete coverage in a pre-commit
scenario, limiting it to the high-value cases would be enough.

I haven't tried to collect data but my intuition is that the most
common breakages are Windows, followed by release builds. The latter
is something people can do themselves (although it's tedious); the
former, not so much.
--paulr

The ARM bots break often enough, too, but recently people seem keen on
fixing the bots quick enough, though, sometimes at the expense of
multiple patches.

I agree even a shallow pre-commit strategy would reduce the noise
levels way down.

cheers,
--renato