[PROPOSAL] Introduce a new LLVM process to resolve contentious decisions

Hi all,

Following up on the extensive discussions since January, many of us would like to put in place a process to improve LLVM’s decision making process for contentious issues. I’ve put together a proposal for how this works, and am recursively using it to get feedback on the process itself. Thank you to the many people who contributed great ideas and improvements during the pitch phases and early drafts of the doc.

Because this is a weird case, I’m not setting up the standard review manager team for this. We’ll wing it, and if it doesn’t work out, we can try again.

-Chris

Evaluation of the proposal:
I fully support this proposal and think it will overall have a positive
impact on the LLVM community.

Related Experience:
None external, but I have been involved in various internal guidelines
for decision making (both creating them and following them). I think
this proposal strikes a nice balance between additional process and
giving members of the community a (time-boxed) opportunity to provide
their inputs. As with any proposal like this, it is likely subject to
change over time, but I think this is a very good starting point.

Involvement in LLVM
Occasional contributor, heavy user.

Self Evaluation
I went through several iterations of the proposal on google docs and
suggested a few changes. I've skimmed through the processes by the other
communities reference in the document to get a high level understanding
of how they work, but have not participated in them first hand. As
mentioned above though, I've been through several different versions of
similar processes through my work and can appreciate the motivation for
this, and see the benefits of much of what is proposed here.

Additional Comments
A few comments on the document:
1. It seems the current document has settled on using threads on
   llvm-dev, however there are still two reference to the LLVM Proposal
   Reviews category on Discourse: last paragraph of Proposed Solution
   section, first paragraph of the Review Discussion Template section.
2. I think it would be nice to try and keep the proposal document on
   GitHub as up-to-date as possible as it evolves based on the discussion on
   llvm-dev. I know this was mentioned and resolved on google docs, but
   something short like: "The review managers and/or authors will update the
   document on GitHub as the proposal evolves during discussions" as
   part of step 7, would be good (IMO).

Thanks,

Kit

Chris Lattner via llvm-dev <llvm-dev@lists.llvm.org> writes:

A few comments on the document:
1. It seems the current document has settled on using threads on
  llvm-dev, however there are still two reference to the LLVM Proposal
  Reviews category on Discourse: last paragraph of Proposed Solution
  section, first paragraph of the Review Discussion Template section.

This was a mistake, fixed.

2. I think it would be nice to try and keep the proposal document on
  GitHub as up-to-date as possible as it evolves based on the discussion on
  llvm-dev. I know this was mentioned and resolved on google docs, but
  something short like: "The review managers and/or authors will update the
  document on GitHub as the proposal evolves during discussions" as
  part of step 7, would be good (IMO).

Agreed - confirmed and added as an example of this. Thanks!

-Chris

Hey,

My 2 cents on this pitch is that I’m +1. In particular, while there are always unknown about how it’ll work in practice and I am curious about what will / will not be using this process (the very first pitch mentioned “this proposal focuses on non-technical decisions”), this proposal is addressing a pain point of the community process.
The process also does not need to be perfect to be an improvement over the status quo, and in the LLVM tradition: it can be incrementally improved as we learn.

Hi Mehdi,

I’m personally also in favor of using Discourse in general, but we as a community haven't converged on that decision. Until a decision is made on that (likely using this process) I think we should stick with llvm-dev as it is the defacto place to have discussions, even given all of the problems it has.

-Chris

This was a mistake, fixed.

I missed that this was changed, I was excited about a Discourse category for this! In particular the second point of the doc points at llvm-dev@ being a problem as the current forum for such discussions.
If Discourse is a no-go (?), then having a separate mailing-list would seem better to me: if only for archiving/searching/tracking such discussions (but Discourse is much better that the mailing-list archives anyway).

Hi Mehdi,

I’m personally also in favor of using Discourse in general, but we as a community haven’t converged on that decision. Until a decision is made on that (likely using this process) I think we should stick with llvm-dev as it is the defacto place to have discussions, even given all of the problems it has.

What about have another mailing-list dedicated for “pitch”/RFCs/community proposals/…?
(It wouldn’t prevent from CC llvm-dev@ when we start a proposal there if visibility is an issue, but I expect the newsletter to help as well

I like the idea of trying to use this process to determine the adoption
of Discourse. I would suggest that another mailing list could be an
alternative to consider as part of that proposal (specifically to
handle the pitches).

Mehdi AMINI <joker.eph@gmail.com> writes:

Thank you to Mehdi and Kit for their feedback on this thread so far - I’d really love to hear from others in the community as well, even if it is a simple “+1 this sounds great” or “I’m concerned about XYZ specific aspect of this” or “-1, LLVM has no problems making decisions” :slight_smile:

-Chris

Hi Chris,

Overall, having a defined/more-formal process to fall back on (rather than start with—a problem with other processes I’ve had to use) seems like a good idea, and what’s outlined here seems basically fine. Two specific comments:

  • Still seeing a reference to discourse (last para of Proposed Solution, after item 9). Thought that should be gone?
  • I thought about the Yet Another Mailing List idea, and concluded that’s not a good idea, unless all RFCs also go there. In practical terms, if you want downstream consumers (or other more casual observers) to notice/track proposals, all proposals need to be in the same place. Whether that’s llvm-dev, or a new “llvm-rfc” list (or whatever), splitting up Where To Look doesn’t help people find anything.[1]

As an aside, I noticed a use of the term “core LLVM contributor” which is not a defined role in this project. I’ve actually bumped into this before, had a panel proposal bounced by a Dev Meeting Program Committee due to lack of “core contributors” signed up to participate. It’s a notion in people’s heads, and probably well defined elsewhere, but is a very fuzzy notion on this project. If we don’t have the role formally, best not to refer to it?

Thanks for driving this, and basically +1 from me.

–paulr

[1] I was once on a project where we had several topic-oriented forums; in particular one call Methods, and one called Designs. People would post process/tip kinds of things in Methods, and design documents/discussions in Designs. Until one day the tech lead decided to post a pointer to his separate design document in Methods, arguing that a pointer to a document is a Method not a Design. I argued (in vain) that having to look in Methods for design info was counter-productive.

So, let’s not split how to find RFCs and how to find Proposals into two separate places, okay?

A part of the problem with “chiming in late” is that people don’t have time to read everything, or even to continuously keep track of what’s going on. I don’t think that people in such position will want to be review managers and take on moderating further discussion. My guess is that they might be inclined to just give up unless the proposal is straight up catastrophic.

On a different note… On more than one occasion, there would be disagreements regarding the interpretation of the official project policies. In the specific cases I remember they related to the downstream/out-of-tree projects. Person A would say “this affects OOT projects”, while person B would say “we should have no concerns about what’s not in tree”. What consequences do you see coming out of resolving these controversies? Would they be “precedents” for the purposes of interpretation of policies?

I appreciate the emphasis on making sure the arguments are laid out clearly, and that the process happens in the open.

The proposal says “A group of either 2 or 4 community members are selected”; the document doesn’t really make it clear who selects them. Reading it a few more times, it sounds like the process is supposed to be “The person writing the [PITCH] selects […]” (and then Chris can suggest adjustments)? Is the person writing the [PITCH] allowed to be a review manager?

I think some broader changes to the LLVM RFC process would make sense, particular in terms of the evolution of LLVM IR. But this seems like a reasonable starting point for ensuring proposals don’t just grind to a halt without any decision being made.

-Eli

Chris,

I am greatly in favor of having a process to elevate issues and settle contentious arguments once and for all. Throughout my career, I often see organizations get bogged down in arguments over the course of years, never solving anything because nobody is able/willing to “have the final say.”

An assorted list of thoughts and feedback items:

  • 2 weeks might be a bit short of a discussion period for major policy changes. Many people go on vacation for that amount of time, and it would be unfortunate to come back from vacation and find that some major decision went in while you were out.
  • I think a separate mailing list would be good for these. RFC’s should also go into this new mailing list, so people who are too busy to follow [whatever]-dev can at least keep up to date on the RFC’s mailing list. I think that it would also be good to have all the subproject RFC’s in once place; more than once I’ve seen an RFC get cross-posted to llvm-dev, then tried to reply and got the “your message is being held for moderation” message because I’m not subscribed to flang-dev (or whatever mailing list). This might also have the side benefit of reducing noise on the subproject mailing lists if RFC’s are not posted (or cross posted) there.
  • Once the process is settled, it might be a good idea to settle some contentious issues that have been argued on the mailing list recently. For instance, the issue of bumping the CMake minimum bound and API stability have been discussed on and off over the last 6 months, with no real resolution. I’m sure there are others that I haven’t noticed.
  • From the FAQ: “Should we improve the existing LLVM RFC processes?” I think this would be a good idea. I think many RFC’s don’t really get discussed until patches start landing, which is really frustrating. If there were a process whereby RFC’s would be approved/denied, I think it would motivate people to pay more attention to them.
  • From reading the pitch, it sounds like the person drafting the pitch is responsible for nominating the 2-4 review managers. Assuming I understand this correctly, I think there are a few issues with this: 1) It might create a conflict of interests. The person drafting the pitch obviously wants it to go their way. They may attempt to (either unconsciously, or consciously), “stack the panel” with people sympathetic to their cause. 2) Similar to how it can be hard to pick code reviewers, it can be hard to know who to pick for the panel. This issue is especially true for new contributors who don’t know all the major players. It might be nice if there were an impartial third party responsible for picking the panel.
  • In a previous round, you posted a google doc instead of a static page in github. I liked this because it allowed for inline comments to be posted and addressed. I think this would be a better format for discussing specific proposals than the mailing lists. If not a google doc (since it’s hard to do versioning there), then perhaps as a phabricator diff, or github PR?

Regardless, thanks for working on this issue!

Thank you,

Christopher Tetreault

My overall impression of the proposal is positive. Like Mehdi, my
thinking is that starting with a *lightweight* process is an
improvement. If the process has bugs, we should strive to fix them.
The process as proposed is lightweight enough that it seems like a
good start. That said, some thoughts:

The proposal does not at all address the problem of total silence.
Some RFCs just don't receive any response, and then people are
uncertain about what that means. Silent agreement? Don't care? Some
RFCs don't come with immediate code changes, so it may not matter too
much -- proposals without code can be difficult to evaluate anyway.
But what about RFCs where the submitter plans to make commits fairly
soon? I tend to feel like acting first and asking for forgiveness
later in such cases, but what's a reasonable time to wait through
radio silence?

I'm looking forward to putting that particular question to the test in
the not-too-distant future :wink:

Other points:

2 weeks might be a bit short of a discussion period for major policy changes. Many people go on vacation for that amount of time, and it would be unfortunate to come back from vacation and find that some major decision went in while you were out.

I tend to agree. While the process as a whole is obviously going to be
longer anyway due to the RFC and PITCH phases, if being inclusive is
part of the goal, then 3 weeks should probably be the minimum on
contentious issues.

I think a separate mailing list would be good for these. RFC’s should also go into this new mailing list, so people who are too busy to follow [whatever]-dev can at least keep up to date on the RFC’s mailing list. I think that it would also be good to have all the subproject RFC’s in once place; more than once I’ve seen an RFC get cross-posted to llvm-dev, then tried to reply and got the “your message is being held for moderation” message because I’m not subscribed to flang-dev (or whatever mailing list). This might also have the side benefit of reducing noise on the subproject mailing lists if RFC’s are not posted (or cross posted) there.

I'm not enthusiastic about this and would rather avoid it. Adding a
new mailing list is potentially acceptable as long as the rule is that
proposals **must** be cross-posted to the relevant project mailing
lists. E.g., if we have a proposals@lists.llvm.org, any proposals
affecting LLVM itself must also still go to llvm-dev@lists.llvm.org.

The thing is, if we do this, then as a corollary the new mailing list
would realistically have to accept mails sent by anybody, not just
people subscribed to that list. This may make having the new list
infeasible in practice.

Cheers,
Nicolai

Overall, +1 from me.

I have a few concerns about the detail, but adopting this with the attitude that we’re expecting to iterate on the process itself seems like a step in the right direction.

I’ll simply list my concerns as I think each of these has been expanded on down thread by others.

  1. Length of review period - two weeks is too short.

  2. Who selects review managers - I’d be fine with Chris having final call. I also think concerns about selection can largely be handled on a case by case basis as part of review feedback.

  3. I’d prefer llvm-dev over another mailing list for now. We could switch later, but I don’t see a strong reason to switch now.

  4. I think it needs to be very clear that reviewers are only expected to have read the current proposal. It should be the responsibility of author to summarize previous discussion. I consider the “this was discussed in X old thread” an anti pattern for this type of discussion.

Philip

Than you all for the feedback, I’m responding to several selected comments in one email with my “proposal author” hat on:

fine. Two specific comments:
Still seeing a reference to discourse (last para of Proposed Solution, after item 9). Thought that should be gone?

Thank you Paul, yes that was a mistake - I corrected this in the writeup.

As an aside, I noticed a use of the term “core LLVM contributor” which is not a defined role in this project.

The intention wasn’t to introduce a new term here, just to make an observation that happens in practice. I’d be happy to change this if you’d like to suggest different wording (or you can directly edit the writing yourself). Thanks!

On a different note.. On more than one occasion, there would be disagreements regarding the interpretation of the official project policies. In the specific cases I remember they related to the downstream/out-of-tree projects. Person A would say “this affects OOT projects”, while person B would say “we should have no concerns about what’s not in tree”. What consequences do you see coming out of resolving these controversies? Would they be “precedents” for the purposes of interpretation of policies?

Yes, I generally consider this process to establish “case law”, where decisions and rationale can be referred back to to help inform future discussions. Nothing is iron clad and immutable though - if we made a decision and it turns out to be wrong or should change in the future, then we should do so.

The proposal says “A group of either 2 or 4 community members are selected”; the document doesn’t really make it clear who selects them. Reading it a few more times, it sounds like the process is supposed to be “The person writing the [PITCH] selects […]” (and then Chris can suggest adjustments)? Is the person writing the [PITCH] allowed to be a review manager?

This is one of the areas that I’m also unclear about. My sense is that someone coming up with a proposal that happens to be contentious will know who the stakeholders are on the “opposition side” because they’ve been the most engaged. I believe that people act in good nature and will do the right thing by suggesting the right people to include, and so it makes sense for the initial proposed list to come with them. This is a “trust but verify” style approach.

However, if this doesn’t work well over time, we can definitely change it!

2 weeks might be a bit short of a discussion period for major policy changes. Many people go on vacation for that amount of time, and it would be unfortunate to come back from vacation and find that some major decision went in while you were out.

I don’t think there is any right amount of time to solve this problem, and extending out to 4 weeks or longer would just lead to unnecessary delays. We should use common sense with this - what I’ve seen happen in the Swift community is some dynamic adaptation. Proposals that need more time get extensions as needed.

I think a separate mailing list would be good for these. RFC’s should also go into this new mailing list, so people who are too busy to follow [whatever]-dev can at least keep up to date on the RFC’s mailing list.

I agree about this as well, and several others raised this. However, I think we should tackle this as a follow-on discussion given the possible move to discourse.

From reading the pitch, it sounds like the person drafting the pitch is responsible for nominating the 2-4 review managers. Assuming I understand this correctly, I think there are a few issues with this: 1) It might create a conflict of interests. The person drafting the pitch obviously wants it to go their way. They may attempt to (either unconsciously, or consciously), “stack the panel” with people sympathetic to their cause. 2) Similar to how it can be hard to pick code reviewers, it can be hard to know who to pick for the panel. This issue is especially true for new contributors who don’t know all the major players. It might be nice if there were an impartial third party responsible for picking the panel.

I agree with your concerns, but I think this is still a reasonable starting point - see the comment above. I don’t think a third party will make it necessarily easier, and an obviously incorrect panel can be objected to by the community.

In a previous round, you posted a google doc instead of a static page in github. I liked this because it allowed for inline comments to be posted and addressed. I think this would be a better format for discussing specific proposals than the mailing lists. If not a google doc (since it’s hard to do versioning there), then perhaps as a phabricator diff, or github PR?

I also agree that google docs are a much better way to do collaborative editing, and I’d encourage people to use them (or any other approach proposal authors prefer) in the drafting / pitch phase of the conversation. However, when the conversation gets “official”, moving to a checked in page has significant advantages - including full versioning, linkability to any revision etc.

The proposal does not at all address the problem of total silence.
Some RFCs just don't receive any response, and then people are
uncertain about what that means. Silent agreement? Don't care?

My sense is that this won’t be a problem if we use the process on controversial proposals. LP-0001 is mostly a clear cut case, which is why the traffic is low. I expect something like “should we move to discourse” to generate a lot more discussion. If it does turn out that a proposal gets insufficient feedback then we can deal with it contextually based on the details of the situation.