Every few weeks, a change is committed to trunk that breaks the AVR buildbot.
A problem presents when commit authors do not fix the build, and just leave it because it passes on the core buildbots. The build stays red for a few days until I go and check it. In the meantime, it likely causes spam for most if not all developers when they commit new code.
All commits should keep master green, but what is the expectation for experimental backends? Is it reasonable to expect all developers who commit code to ensure tests pass on the AVR backend?
On top of this, is there any way to notify maintainers of a backend when a buildbot has been failing for some time? I imagine other experimental backends have run into the same problems.
Every few weeks, a change is committed to trunk that breaks the AVR buildbot.
A problem presents when commit authors do not fix the build, and just leave it because it passes on the core buildbots. The build stays red for a few days until I go and check it. In the meantime, it likely causes spam for most if not all developers when they commit new code.
All commits should keep master green, but what is the expectation for experimental backends? Is it reasonable to expect all developers who commit code to ensure tests pass on the AVR backend?
Hi,
The builder isn’t marked as experimental so I think the expectation is that people keep it green and contact the bot owner if they need help figuring out why their change makes it red. That said, it sounds a bit odd to have a non-experimental builder for an experimental backend.
On top of this, is there any way to notify maintainers of a backend when a buildbot has been failing for some time? I imagine other experimental backends have run into the same problems.
I used to have all the MIPS buildbots send me an email on every failure and filter those emails into a folder. I’d then use the unread mail count to keep track of how long it’s been failing and investigate if it was more than the occasional build.
this probably depends on what we want experimental targets to be and how
they are different from normal targets. From my developer perspective,
anything that is not enabled by default and is not checked by default in
a normal LLVM build is something a normal developer should not need to
worry about. As such, I do not
expect committers to investigate bugs triggered in the AVR backend. If
we would believe the AVR backend is stable enough, such that users can
rely on it and such that other developers are unlikely to trigger bugs
in the AVR backend, the AVR backend should most likely be promoted to a
stable backend.
As a result of this, I would also expect buildbots of the AVR backend to
not send any emails to the general public, but to instead send emails to
the buildbot owner and maintainer of the AVR backend.
In my perspective, it is the job of the buildbot owner of an
experimental backend to investigate failures in this backend and to
clarify if a failure is caused by a bug in a recent change or rather by
a bug in the AVR backend.
If it seems as if the original change committed is incorrect, the
buildbot owner should work with the committer of the corresponding patch
to get the bug resolved.
Obviously, the answer I gave above is very black and white. In reality,
the real answer depends on the maturity of the backend and the quality
of the availability of the buildbot owner. What I wrote above has a
rather new and unstable backend in mind -- especially the category of
backend we would expect to become experimental in the first place. The
more stable a backend is, the more rare breakages are, the faster bugs
are fixed, and the more likely it is that breakages are due to bugs in
LLVM commits, the more it makes sense to have a buildbot that sends out
emails to the actual committers.
In Polly I have two kinds of buildbots. A set of buildbots which are
experimental and do not send emails ever and a set of bots for which I
know they have a very low rate of failures (every couple of weeks) and
for which I also make sure that I myself can either fix or XFAIL any
issues very quickly (commonly at most 2-3 failing builds during working
hours). I do not expect anybody to fix failures in polly. However, I
made the experience that people are kind enough to help fixing problems,
if the bot generally has a reputation of being green most of the time.
In your case I suggest to make the buildbot experimental, disable emails
to committers and add yourself as email receiver. When you can make sure
the bot sends very rarely emails and you yourself can make sure bugs are
addressed / XFAILED within a very small delay and this has been proven
to work for a couple of weeks, you could carefully consider of enabling
emails to other again.
If I was in this position, I’d also configure the bot to build only the AVR backend. That’s help make sure that an email does get send when a test fails in the X86 backend.
The builder isn’t marked as experimental so I think the expectation is that people keep it green and contact the bot owner if they need help figuring out why their change makes it red. That said, it sounds a bit odd to have a non-experimental builder for an experimental backend.
I see. I had followed the generic How to add a builder docs, which doesn’t mention the concept of an experimental buildbot. I’ll send a patch to mention it.
If we would believe the AVR backend is stable enough, such that users can rely on it and such that other developers are unlikely to trigger bugs in the AVR backend, the AVR backend should most likely be promoted to a stable backend.
In general, I’ve found that almost all of the time that the AVR build breaks, it’s been something pretty small which also caused a bunch of other targets to fail also, which I suppose is a good sign. On the topic, I plan on following up on promoting the backend to stable once the current effort of enabling AVR in Rust is complete and we’ve ironed out any bugs found in usage.
As a result of this, I would also expect buildbots of the AVR backend to not send any emails to the general public, but to instead send emails to the buildbot owner and maintainer of the AVR backend.
That sounds good. My plan is to make the buildbot a staging bot, and then be the sole receiver of emails from it.
If I was in this position, I’d also configure the bot to build only the AVR backend. That’s help make sure that an email does get send when a test fails in the X86 backend.
I would love to do this, but there’s a bug in the backend which causes a few of the Generic CodeGen tests to fail. To work around this, I leave X86 as the default target for now. I’m definitely planning on updating this once I’ve fixed the bug.
This usually happens when LLVM_DEFAULT_TARGET_TRIPLE is not explicitely set and you end up with your host machine as default while not building the x86 target. If you set LLVM_DEFAULT_TARGET_TRIPLE to some AVR ones the failure should go away (otherwise complain and file bugs).
When I make AVR the default target triple, most of the generic CodeGen tests pass, but some don’t. The easiest solution was to make X86 the default target so the generic CodeGen tests run under it, and that way the AVR specific tests still run.
The bug causing the tests to fail is not because the default target hasn’t been built, it’s an assertion error hit inside the AVR ISel code.
Once the generic suite passes, I will make AVR the default target on the builders.