Merging PRs that have a "Changes Requested" annotation when the reviewer is away

Hi all, I’m looking for some general thoughts from the community about something that has recently happened. To be clear, I do not believe anybody has done anything maliciously or anything, but I do think it would be helpful for people to be on the same page as to what is/is not appropriate, in the view of the community.

I’ve been involved with reviewing https://github.com/llvm/llvm-project/pull/157499, authored by @Prabhuk and with @ilovepi having reviewed and approved much earlier in the process. @ilovepi asked me to review the patch before it landed, which I duly did and marked it with “request changes” so that it was clear the PR wasn’t in a sufficient state to land it at the time. We continued with back and forth reviews and updates up until mid December when I went away on my Christmas break. I didn’t get back until the start of this week and have been trying to catch up on a large email and review backlog in the meantime.

@Prabhuk addressed my most recent round of comments not long after I finished work for the calendar year, then pinged a couple of days later (aside, not the usual week period that is recommended). A week ago, they then, with no further input from anybody, posted a comment saying they’d merge the PR soon, because it was blocking downstream work. They merged the PR about a day later. In doing so, they stated they’d be happy to revert etc post landing, if I came back and had any further concerns.

My main concerns here are twofold:

  1. Although @ilovepi approved the PR a while back, it had changed a fair bit since the original approval, so I think it could be argued that this PR was effectively landed without explicit approval, especially as I had clearly raised concerns after this approval.
  2. The idea of saying “I’m going to merge this due to a lack of response and it’s blocking me” is a bit of a dangerous idea and seems to fall afoul of our guidelines here.

Are my concerns legitimate, or am I viewing things from the wrong way?

What I think could have been done better:

  1. I could have communicated my expected absence period over the Christmas break on the review.
  2. @Prabhuk could have sought review from another developer, e.g. @ilovepi to confirm that all my points had been addressed.
  3. A specific note of a need to get the change landed might have prompted me to suggest whether another reviewer could finish things off in my absence.
5 Likes

Particularly given the state of reviews to reviewer ratio, I usually do not have an expectation that someone leaving a review is going to ever come back to re-review it after a certain period of time. When I give reviews, I have even less of an expectation that I will come back to re-review it. Others can continue with the review and approval.

I do not view “requested changes” as a blocking status. When I (rarely) remember to use it, it’s mostly as a way of getting it out of the list of PRs waiting for my review. Merely commenting doesn’t qualify as a GitHub state change to move it out of the list of PRs I should be reviewing.

1 Like

I agree, I think your concerns are completely legit here.
Rushing anything because of downstream concern does not look like a good practice to me.

3 Likes

Lack / absence of reviewers sounds like a case where I would reach out to the maintainers of the component.

3 Likes

To me this is the key information.

If I was the author, I’d have sought your feedback directly (discord, discourse, irc, smoke signals), but also other people (primarily other reviewers) to “make sure I have addressed the original concerns”.

Key issues I see:

  • @ilovepi had extensive reviews and approved when the only concern you had was some extra header. But he also explicitly mentions to wait for your review.
  • You have iterated through extensive reviews, which the author has marked as “done” and you identified as “definitely not done”. This is a key concern of the ability of the author to identify what tasks were actually done and could be closed.
  • @ilovepi continued to review, and when you “requested changes” you explicitly said “there’s still plenty of work to be done”. This was almost 2 months ago.
  • The author was on a break and came back 20 days after your comment, and after some updates, you continue to review.
  • Author marks many conversations as “done”, but there’s no feedback form you if they were actually done.
  • Pings 19th Dec.
  • Merges 7th Jan with clearly invalid comment: “This patch not being landed is blocking us on a couple of fronts downstream”.

To me, this is a violation of our PR ethos and behaviour, but not directly our current policy. There’s wording on requesting changes, and pinging a PR, but nowhere we allow someone to just merge without acknowledgement of a reviewer that requested changes.

While there was a long time between the ping and the merge, there was also a long time between many interactions in that PR, hinting that the “urgency” wasn’t there until the merge.

We already have a “please don’t block PRs”, so we may also want to add an explicit “please don’t merge with blocking reviews” and also “how can reviewers unblock reviews with unresponsive reviewers”.

Ultimately, we have ways of escalating to maintainers, area teams, etc. which PR authors can make use in case someone is unavailable for an extensive period, so to me, merging with a downstream excuse was clearly the wrong move.

5 Likes

FWIW, I am the main maintainer of this particular component. There’s one other (@MaskRay), but he’s not as active as he has been in the past due to other commitments.

2 Likes

I agree that the status alone doesn’t qualify. That’s why I said it’s not “directly violating policy”.

What I see as blocking in that review is not the status itself, but the repeated and abundant signals by the reviewer (which happens to the be maintainer for that part of the code) that the PR is not ready for merging.

What I see as violating the policy is that the reason behind the unilateral merge was “this is blocking my downstream”.

Bonus points for that changing the interface of a key LLVM tool so close to the release.

IMO, we should revert that PR from both main and release branches.

3 Likes

Apologies once again for breaking protocol with this patch. With the communication I provided in the PR I thought landing this PR would be frictionless. But looking at the feedback from @jh7370 and the rest of the posters here it apparently was a mistake. I do not want to speak anything towards the larger protocol for the community. Just want to get the issue of this PR sorted. Here is what I think I am suggested to do based on the discussion here. Please correct me if there is anything I am missing here:

  1. Revert the patch from main and release branches.

Going forward I will make sure that I escalate through appropriate channels for further review when I am blocked in such a situation. I am out from 16th January for about 10 days and would like to sort this out before I head out. I will wait for @jh7370 to confirm if this course of action settles the issue or if he sees it differently.

4 Likes

Thanks! I’ve also left some comments on https://github.com/llvm/llvm-project/pull/157499 , which you can address after your are back :slight_smile:

1 Like

Thanks @Prabhuk. I started looking at the PR this morning, but didn’t get particularly far and it’s not just a case of a few minor things to tweak, I think, so I think revert and new PR is the best approach here.

On the note of “we need this to unblock downstream stuff” do you not have a private fork of LLVM downstream? At least in the Sony toolchain, we have an internal fork of LLVM with plenty of private patches. If we needed something for a particular deadline and couldn’t get it through quickly enough upstream, we’d land it in our downstream fork, then look to upstream it when time allows.

2 Likes

+1. This is the right way of using upstream if you have deadlines that need to be met. This patches can become PRs or even in parallel with an upstream PR, and then revert before rebase, etc.

2 Likes

Yeah, I see this as a problematic approach and do not think it is a community best practice. It’s one thing if the build is completely broken and we need to get it back to green. But for regular work (features, bug fixes), that’s not acceptable IMO. Especially around the branching timeframe. (I concur with @rengolin about reverts being appropriate right now; not that I think anything is malicious or done in bad faith, but we’re in the process of trying to get rc1 out and these changes did not undergo appropriate review yet.)

Personally, I think “Request Changes” means “I do not want this to land until I’ve re-reviewed it”, and if that takes time, that’s still okay. Obviously, if someone marks request changes and then disappears for months, that’s a different situation. But for active reviewers, I think the usual expectation is that a “request changes” is a blocking signal. But I don’t think we document this anywhere and so I’m sure others have different opinions.

3 Likes

Reverted on main. Back port on release branch waiting to be merged here:

Since the downstream part about my comment had caused some stir I’ll try to address it. My justification for landing the patch at that time was because I felt all the comments raised were addressed sufficiently imo, no explicit reason not to merge according to policy and did not have any clarity on when the reviewer may return from their break. Given my break was around the corner as well I wanted the patch to have some soak time after merging before my break began. I’ve already conceded I should not have merged. While unblocking downstream work was also a concern that was not the primary motivation for merging and I should’ve made that clear in my comment. My focus at that time was to communicate that I will be open to reverting the patch if the reviewer was unhappy when they returned and did not think about much else.

Hope this clarifies the original intent and addresses the problem at hand.

1 Like

I think it was clear pretty quickly that this was not intentional, but thanks for addressing it directly. My points above were only about making these points clearer, perhaps changing the policy wording.

It may not be common in other projects, but in LLVM it’s very common for reviewers to do “drive by”, partial comments, and leave. This may give the impression that they’re not coming back, or that the review is finished. Sometimes it actually is, so it’s not always clear.

When I do that myself, I normally say “drive by comments” if I won’t come back. Or if I will, I say “I will look at X later”, which means I want the author to wait until I can look at that, or if I “disappear”, others can look at X and do their review. I’m happy for others to look at that and even approve, I just want to make sure “someone” does.

But when people actually click the “red button” or say things “this is not ready to merge”, they’re raising the bar for merge. I think we should make that clearer on the policy from now on.

Another thing that has been discussed a lot recently (on the GitHub threads) is “who marks the conversations as done”? If we enforce reviewers to do that, even the most silly comments will require them going back to every “drive by” comment. If not, then how do authors judge if it’s really done?

Rule of thumb is, if the comments are nits, formatting, slight changes, then authors can easily close them. But if a significant change was requested, especially when someone has raised the bar, then I’d ping the reviewer to double check before merge.

In the absence of strict rules, these situations will continue to happen, and when they happen, we just clarify on both sides and carry on, like we just did. :slight_smile:

4 Likes

As a very rare/peripheral contributor, I found this paragraph really informative and thoughtful. I’d love to see rules of thumb like these documented, or at least made more visible if they already exist. I found the discussion positive and helpful from all sides, and I don’t see it as “someone did bad,” but rather as “how do we build the best possible community and product given limited time.” I’ve been confused by this in the past due to expectations shaped by very different, fully paid corporate environments. Respectful discussions like this are exactly what will preserve LLVM’s amazing and productive culture.

1 Like

FWIW, I actually ask authors of PRs who are responding to conversations I’ve started in a review to never mark as resolved, because I’ve found even sometimes trivial items aren’t addressed correctly by some developers (you can see a discussion of this in RFC: GitHub PR "Resolve Conversation" button ). There didn’t seem to be a real consensus in the conversation, so it’s not in the LLVM docs on contributing and GitHub reviews etc.

3 Likes

Thank you! FWIW, I didn’t think this was malicious or anything particularly negative. Frankly, this was post-commit code review working as intended – someone saw something after the commit that raised questions, we discussed it, decided on a course of action, and went on with it. I think everyone was acting in good faith and we ended up with a good outcome!

3 Likes

I’d probably prefer the heuristic to be:

If the suggestions I made are “fire and forget” (ie: I trust the PR author to be able to address them - and/or that the impact of them failing to address them to my satisfaction is small enough that I’m happy to deal with them post-commit) then as a reviewer I’ll approve the patch along with the feedback.

If my suggestions are still “fire and forget” but I’m also not ready to approve - I should say that explicitly (opt-in to allowing other people to approve without coming back to me). I believe the current docs suggest this (oh, I think perhaps it used to be that the author was responsible for this, but the docs now suggest the reviewer/approver is responsible for this, which is fair (they’re more likely to be able to make this judgement accurately)): “When providing an unqualified LGTM (approval to commit), it is the responsibility of the reviewer to have reviewed all of the prior discussion and feedback from all reviewers ensuring that all feedback has been addressed and that all other reviewers will almost surely be satisfied with the patch being approved. If unsure, the reviewer should provide a qualified approval”

So, yeah, I guess in this latter case - even if I don’t say it explicitly, if the feedback I’ve given has been addressed sufficiently that some other reviewer thinks I would agree to it, then they can provide unqualified approval. (if I’d done some trivial review, but expected to come back for more in depth review and that wasn’t otherwise fairly likely - I should say that explicitly)

(I think eventually we’ll need to move to a model with more explicit ownership/approval authority - I don’t think it’s practical for any random person with approval ability to know what kind of depth of review other reviewers want to give, or to explicitly request it in every review, and in the absence of that small cliques can form without much guidance/oversight)

1 Like