RFC: Maintainer Roles and Beyond

Following our effort to establish and update maintainer lists, and multiple discussions in the Project Council, we would like to ask the community for feedback on the perception of the maintainer and lead maintainer roles as well as whether and how they may be complemented or clarified.

This is not a policy change proposal but a request for comments, contributions and community brainstorming.


  1. Do we understand “maintainer” as a mostly organizational/managerial role (triage, mediation, information forwarding) or as a technical leadership role (keep consistent technical direction, influence design)?

Our developer policy states that

[…] the project needs volunteers willing to do the less glamorous work to ensure we produce robust, high-quality products.

Maintainers are those volunteers; they are regular contributors who volunteer to take on additional community responsibilities beyond code contributions. […]

suggesting that maintainers have the same rights as any other contributors but more responsibilities (* interpreting the semantics of “regular” as “normal” rather than as “more than occasional”). This is reinforced by further wording:

All contributors with commit access to the LLVM Project are eligible to be a maintainer.

This is reflective of our open, community-driven governance approach but less common than the stance many other open projects adopt, Linux and Pytorch come to mind as example, where maintainers play a more significant role in driving the technical direction of the project and deciding which contributions will be accepted.

There is currently no intention to change towards such a contribution model.

However, this creates a confusion, not always limited to occasional contributors or folks contributing to multiple open projects, as to whether LLVM maintainers have more rights (up to a veto right on contributions) or otherwise carry heavier weight in community decisions. This is reinforced by the listed responsibilities of a maintainer:

  • ensure that commits receive high-quality review, either by the maintainer or by someone else,

  • help to confirm and comment on issues,

  • mediate code review disagreements through collaboration with other maintainers (and other reviewers) to come to a consensus on how best to proceed with disputed changes,

  • actively engage with relevant RFCs,

  • aid release managers with backporting and other release-related activities,

  • be a point of contact for contributors who need help (answering questions on Discord/Discourse or holding office hours).

that effectively require a maintainer to be a proven expert in a field, which implicitly gives them a stronger voice.


  1. Do we understand “lead maintainer” as covering “everything not covered by somebody else” in the project (historic description of Chris L. in the maintainers document) or as covering everything in the project, potentially having more authority than regular maintainers of the same project?

The policy is rather terse on this:

This role is like any other maintainer role, except the responsibilities span the project rather than a limited area within the project. If you cannot reach a maintainer or don’t know which maintainer to reach out to, a lead maintainer is always a good choice to reach out to.

This is somewhat different from point 1. We have two major, actively maintained projects – LLD and MLIR – that are currently lacking lead maintainers despite having “regular” maintainers in each area: different targets for LLD, explicit categories for MLIR. It has proven challenging or contentious to find lead maintainers for those given the breadth.


  1. Should we have a separate, purely organizational role in addition to or instead of lead maintainers?

One of the plausible ideas is to introduce a separate role, tentatively called “point of contact” for the project, that would focus specifically on the communication aspects of the current maintainer role: add the right people to review, ping them to comment on RFCs, tell contributors who they should contact further for a specific question. This is more of an organizational role that requires less technical depth and more breadth.


  1. Do we need lead maintainers if all of the code in a project is already covered by someone?

Another plausible idea is to relax the requirement for lead maintainers as long as all subdirectories of a top-level project are covered by some maintainers. This would “resolve” the situation with LLD and MLIR, but is more difficult to keep track of in the longer run, and potentially harder for contributors to navigate.


You can also find record of the Project Council discussions in the document:

we several sessions last year where the topic was discussed.

All sorts of suggestions and comments are highly appreciated. cc @project-council

3 Likes

I understand it as being both; you are responsible for triage, mediation, reviews, information forwarding, etc but you’re responsible for those things because you are a technical leader in that role. e.g., we would not accept someone as a maintainer who has never submitted a PR or done a review in the area they’re volunteering to maintain.

I understand it as having the potential to have more authority than regular maintainers of the same project, but that authority should not be exercised except as a last resort. e.g., if you have two maintainers disagreeing about a technical change, I think the community expects a lead maintainer to step in and help resolve the disagreement. But I think the community expects “resolve the disagreement” to be about consensus building rather than making decisions by fiat and decision-by-fiat is a last resort for when we need a decision and it’s clear consensus is not to be found.

Yes please! As an example, I am the lead maintainer for clang-tools-extra. However, that’s a very broad project with lots of unrelated parts in it, so I don’t know the technical details of everything in the repo and that means I feel kind of uncomfortable being called the “lead maintainer” for it in some ways. Thankfully, each of the active projects has maintainers signed up for them so it’s not a huge issue in practice. You could imagine that project having no lead maintainer, but it’s helpful to have a single point of contact for the repo because there are things like maintainer list refreshes, questions about adding new projects, etc. Having the notion of a “point of contact” for the project makes a lot of sense to me because that seems like a better description for the role in that case.

Yes*. I think we need a list of people we can reach out to for “project level” topics. For example, one of the roles of the project council is to ensure that the maintainers lists across the entire monorepo are up to date. When I was doing that effort last year I ran into questions of “who do I talk to” for things like lld and mlir because there are active maintainers but nobody I can go to and ask to refresh their maintainers list. But I think “point of contact” would suffice for this sort of thing and so we don’t need a lead maintainer specifically.

2 Likes

I think the current policy phrases this as the former, but the reality is closer to the latter. In general, I don’t think the set of people that review any particular area is large enough for there to justify a dedicated management role. The exception might be for functionally unmaintained components, where a non-expert could volunteer to pull in maintainers from tangentially related areas.

1 Like

Thanks for putting together this RFC. It’s something I’ve been thinking about actively since I became the lead maintainer for LLDB.

I concur with Aaron and Matt that the policy says the former, in reality it’s both, probably leaning more towards the latter. Speaking for myself, my motivation for being a maintainer is the technical leadership. I’m invested in the health of our sub-project and the organizational responsibilities are a means to achieve that.

As pointed out, this already happens implicitly, and I don’t think we’re not doing anyone a service by pretending otherwise. I’d prefer if the policy matched the reality (rather than the other way around).

I’ve always considered it to be the later. I don’t know if it comes with more authority, but I definitely think it comes with increased expectations. I see the lead maintainer as the last defense against something falling through the cracks.

I expect every reviewer to consider the impact of a change on the project as a whole. However, I consider it the lead maintainer’s responsibility to make sure every part is covered. For example, if there’s a PR that touches support for say BSD, I’m counting on the respective maintainer to review that part, but I don’t necessarily expect them know who else to add to cover everything else.

That also answers question (4): yes, I think it still makes sense to have a lead maintainer when the part of a project are fully covered.

If we change the meaning of a maintainer, I think it makes sense to have a more accessible role. I welcome anything that helps facilitate reviews.

1 Like

To +1 on what others have said, I’ve sometimes felt like the written policy was not entirely reflective of reality with respect to the role of maintainers. In my experience, people actually expect that a maintainer also has some amount of technical leadership, and technical leadership is required for any project to thrive anyway.

In practice, I’ve found that being a maintainer means acting as a consensus builder and facilitator, but also a “memory” and “source of inertia” (not in the negative sense) for the project, which is important to retain a coherent long term direction. In other cases, it means taking responsibility for a situation (e.g. a bad bug in a release) and handling it as if it were your own, even if it might not have been caused by you. And in a few cases, it also means taking a stance because someone has to. De facto, I think the role is currently one that includes technical leadership, and while I don’t mind the current wording, I think it would be better for it to reflect reality more closely.

4 Likes

To echo what everyone said, but from a non-lead maintainer perspective, I would not want e.g. Aaron to feel confined by the current wording of the policy and stop providing technical direction, resolve disagreements, and speak in conferences and committee meetings with his lead maintainer hat on simply because it’s not acknowledged by our policies.

1 Like

Both, at least to me.

Maintainers are expected to contribute to the project over the long term, both through code and through reviews. For a regular contributor, it is perfectly natural to move between areas, make a few casual contributions, and move on. A maintainer is in a different position: they are more likely to think about the long-term trajectory of the code, consistency of design, and the impact on future contributions.

However, despite these extra responsibilities and expectations, there does not seem to be much explicit mechanism to support them. That is the main gap that I see. If maintainers are expected to think long-term and help steer things, then it would be good to also define how they make decisions and how disagreements are escalated.

For example, PyTorch’s governance docs make this kind of support much more explicit:

Each maintainer group may adopt its own rules and procedures for making decisions (majority vote being default).

or

In the exceptional cases where module maintainers cannot come to a conclusion themselves, they will escalate to core maintainers for review.

I am not saying LLVM should copy that model directly, but I do think LLVM would benefit from being more explicit here.

This is a bit tricky for me, as we do not have lead maintainers in MLIR. Also, MLIR is a bit unusual in that there is “MLIR core infra” and then there are upstream dialects. To me, that feels a bit like “Clang” vs “clang-tools-extra”.

Going back to the earlier point about maintainers’ extra responsibility: what should the escalation path be if maintainers disagree? Should that go to lead maintainers?

That already seems at least partly reserved for area teams, from LLVM Project Governance:

(…) area teams are responsible for facilitating decision making for their area of the project. Facilitating decision making can take any number of forms ranging from contributing to RFC discussions, helping mediate disagreements, or fulfilling roles originally delegated to Chris Lattner in the LLVM Decision Making process.

If disagreements are meant to be resolved by area teams, then I would expect lead maintainers to focus on something else. Or perhaps area teams could decide to delegate some of that responsibility to lead maintainers.

Either way, with the current wording, I do see scope for overlap between “area teams” and “lead maintainers”, and I think it would be good to clarify that boundary explicitly.

I am not sure we need a dedicated new role for this.

That said, I do think there is organizational work that needs doing, and I would personally like to see us leverage GitHub more to organise and categorise PRs and issues. So even if this does not become a separate formal role, somebody still needs to drive that work.

IMHO, no.

But I do think we need clear escalation and conflict-resolution paths. To me, that is the more important missing piece.

-Andrzej

1 Like

With the caveat that this technical leadership is about consensus, not dictating a particular direction. LLVM is a very diverse project. We have users from all over various spectra that are also experts in their fields, and their voices should not carry less weight. LLVM’s own policy is very clear about this.

So, worry and be proactive about direction? Definitely. Push for a particular direction over others, alienating developers and other maintainers? Definitely not!

I think MLIR’s lack of “lead” maintainer is a good thing. We have maintainer to cover everything already, and we’re all “experts in our own fields” already. One single voice over the existing maintainers would not add anything, but could dampen the existing voices of the current maintainers.

In a way, MLIR has multiple lead maintainers, if you think every group maintainer to be a lead maintainer, too.

That’s the job of the area team, tbh.

Area Teams?

Agree.

Er… Area Teams?

We don’t have area teams for everything though. e.g., lld is three separate linkers in a trenchcoat but has no area team, compiler-rt handles a bunch of unrelated things (sanitizers, builtins, runtime ABI layer, etc) and no area team.

FWIW, this was a real problem I ran into as chair of project council last year – there are parts of the monorepo where I wanted a clearly-identified person (or people) to reach out to for things like maintainer list refreshes. But we’ve also had issue triage needs, questions on discord, etc; I don’t think an area team is the appropriate venue for all of that.

2 Likes

Thoughts as the LLVM-libc lead maintainer

I’d agree with the others and say there’s some of both. I like Aaron’s comment about maintainers being a point of contact. If you are maintainer for an area and someone reaches out to you about making a change you should be able to make that decision, either by coming up with a technical solution or by reaching out to someone else.

I think that lead maintainer does implicitly cover everything in the subproject for the purpose of mediation. Other people have brought up area teams as an escalation path but there are no area teams for any of the runtimes.

I don’t think that LLVM-libc needs one, but we’re still relatively small. We’re at 12 active maintainers by my count. That being said for projects without a clear leader I think this makes sense.

I’d say having a point of contact is useful, as others have already mentioned. It doesn’t need to be someone who is “lead maintainer” though. I also agree with @banach-space that having a clear escalation path is important, regardless of what you call it.

Area teams are not (currently) the venue for all of that, in the same way the existing maintainers aren’t either. Creating yet another hierarchy will take care of the immediate problem, but we’ll always leave holes in the structure, XKCD style.

Back before all the changes, Chris was the catch-all, so we didn’t have to worry about it. Now, it seems we’re fixing it by creating too many hierarchies. I’d pause the proposal of new structures and ask what do we have, what do we need and what do we want.

If we want technical direction, then we need more than one “lead” to make sure we’re honest to the actual community and do not concentrate actual power into a single individual. This also helps share the load, vacation, etc.

If we want representation, we need elections, and that’s the area teams. But we can’t have these people to dictate technical direction either. Popularity isn’t always correlated with long term investment or real stake in the project.

If we want coverage, then we need to either find more people (very hard) or accept distant connections. And if we accept distant connections, we need to accept that some maintainers, even some lead maintainers, will not be as relevant as others. This has always been true and well known in LLVM, but it seems people want more clarity there.

So I’d propose we decide what we want, we have one structure (a mix of maintainer/area teams is fine) to represent it and we pay the cost of having that structure. The old LLVM way that we can pay no structure cost is generating more problems in the past 5 years than it passively dismissed in the previous 10.

But there’s not a proposal to create more hierarchy here? This is my take on the RFC:

There is a problem: we need a clearly defined escalation path for every part of the monorepo so when there’s a question, anyone can figure out “who do I go ask for more help?” (“Escalation” does not mean “dispute resolution” in this case, I just mean “I need to talk to someone about something in this area”.) Our current policy says that every part of the project has a lead maintainer but the project council recognizes that doesn’t reflect reality. So do we 1) update the policy to not require everything to have lead maintainers, 2) add lead maintainers to everything, 3) come up with “point of contact” and clarify the policy to say that every project has lead maintainers or points of contact. 4) something else entirely.

2 Likes

So far, most discussions on “lead maintainer” have led to the idea of technical direction more strongly than just a point of contact. So I’d be wary of adding lead maintainers to everything. We also haven’t agreed on what does it mean to be “lead”. For example, MLIR has “group maintainers” for the three groups and that should encompass the whole project, so we don’t need another level, but we also don’t have a nominated “lead”. I’d personally be against (2).

Though, if we choose (1), then we’ll have to choose (3). I propose some ideas for (4) below.

I think “escalation” is a loaded term that we perhaps should avoid.

I think that’s an outdated notion, both in the need to have one and in that being one (“a lead maintainer”). So at the very least I’d change that wording. We have discussed this multiple times in previous RFCs and council meetings, so it’s not something new.

I would personally attack this problem from a “least effort, least impact, least surprise” principle. We have handled maintainers like that before, so it’s also not something new.

  1. What are the projects that are “in trouble”? Is lld’s lack of maintainer impacting new contributions, support quality or something else? If so, then we try to fix lld. Repeat analysis for any other project that needs it.
  2. What are the current maintainers that feel uncomfortable in their overreaching? Is your maintainership of clang-tools-extra a burden, or do you worry you provide a sub-par service? Then talk to the clang-tools-extra’s community and find a better suitor.
  3. Can we have a catch-all service instead of multiple catch-all people? Using generic/group handles for pinging on PRs and RFCs, or a forum/github mechanism to reach out multiple people that are willing to help, but don’t necessarily need to be tied to a single project or have any technical expertise in the area.

My point was just to avoid loaded terms, cross wires with existing roles, or creating new roles that make our lives harder with limited benefit.