Hello everyone,
Since we moved to github PRs and people are building their workflows more github centric, I’m going to make all the production build bots (except any with the tag "silent) reporting build statuses to github.
We have this for few selected builders for years. Now is the time to make it for all.
Thanks
Galina
That sounds very useful, thank you.
How will this reporting work?
I can imagine a few different ways it could work.
I’m guessing one potentially useful way would be for a bot that goes from green to red to post on the corresponding PRs that it potentially found an issue with that PR? But maybe that would produce too much noise on PRs that do not have an issue and that happened to be tested together with the PR that triggered the regression?
Thanks,
Kristof
IIUC, since our current production bots are post-commit testing, this isn’t about PRs, but commits. If the PR squashes, they’ll see that as a single commit, if not, they’ll test all commits (atomic merge) and mark each one individually as green or red.
You can see that on the commits list from the main page. If you click on the status icon it expands to all builds that reported status (and usually have a link to the particular build).
How to use that information is a different topic. I imagine there are plugins that look into breakages, ranges, when they were fixed, probably even help you bisect builds, etc. But that’s a separate (and more complex) matter.
Initially, just having the green/red per build on each commit would be awesome in itself.
Thanks for answering, Renato!
Build statuses are reported for a particular commit to the main branch, which was built and tested. If a bot didn’t build a particular commit, it does not report the build status for that commit. Exactly the same way how it works now for, let’s say, llvm-clang-x86_64-expensive-checks-ubuntu or clang-x86_64-debian-fast builders.
“Details” link leads to that build, so one can check build steps and logs for details. We keep build history for about 1 moth.
Oh that would be awesome if buildbot could do that for commits which are referencing the #12344 PR number and that a built alone. Much useful that the build status for developers!
It’s also likely not very difficult to build, if we already have the ability to call the GitHub API, we’re only a few lines away from getting there.
The only use I know of build-status is for automation trying to find a “known good” tested build (and even then it’s iffy with some of our bots because of batching: not every commit gets a status, so two bots could only rarely test the same commit in theory)
That would be for the purpose of the history only, right? One cannot reopen already merged PR, and cannot revert the PR from github UI.
Anyway, I like the idea. I’ll see what could be done to comment PRs in addition to reporting build statuses for commits.
I was mostly thinking about reviewers notifications: the person who click “merge” today would not receive the buildbot notification if they aren’t the author (right?) and the author may not know what to do if they aren’t a frequent contributor (and without commit access they can’t revert anyway).
Actually I see a “revert” button for a merged PR in the UI:
But you can’t “reopen” indeed.
This does not work. If you’d click on it, you will receive a 500 error. I’m currently trying to figure with GitHub what is the issue.
Buildbot includes to the blame list both, the author and the committer. So, it should work modulo unknown issues.
If we comment on PRs, people would receive 2 notification on each failure - one from the buildbot, and the other one from github PR. I don’t think this is a big dial, and since PR would be commented only on selective failures, when a build goes from green to red on a single commit, the notifications from PRs would add some value. We will see how it will fly.
Something that’s unfortunate about reporting all statuses to GitHub is that it’s no longer possible to see whether the build is currently “broken”. We pretty much always have at least one failing buildbot at any given time, so all commits will be marked as failed. Previously the status was a pretty good indicator of the state of the tree, i.e. you could see whether it was severely broken (fails to build, fails tests, fails expensive checks) without caring about failures on something as exotic as bolt-aarch64-ubuntu-clang-shared.
@nikic Not sure I follow. Are you saying that bolt-aarch64-ubuntu-clang-shared is not reliable and reports false negatives? Or just that you do not care of that particular build configuration?
We have a common build tree (aka monorepo). So, if any project or a runtime, which is a part of that common build tree, is broken then the whole build tree is broken. From that perspective, I’d say, currently we have more accurate build status for particular revisions. If it is red too often, and enough people want to see it mostly green, then something should be done about addressing issues faster. On the buildbot side we can be more aggressive with flaky builders if any, and builders which do not get attention.
Buildbot includes to the blame list both, the author and the committer. So, it should work modulo unknown issues.
[/quote]
Actually it was mentioned here that when merging a PR for someone else, GitHub does not propagate the information about the person who merges. So right now I don’t think we notify them right?
Yes. In this case we notify noreply@github.com.
I’ll think of adding an exception to skip that in the blame list, as there is no point of sending notifications there.
I was bringing this up because this motivates again the benefit of notifying inside the PR directly, doesn’t it?
Yes. And that’s work in progress.
