Proposal: an anonymous maintainer workload survey

LLVM Project
1

In the AI-generated communication thread, Aaron mentioned that a long-time maintainer is close to stepping down, and that this is not the only case. I’ve heard similar things.

I propose a short anonymous survey for people listed in the Maintainers.md files, run by the project council (a neutral party rather than an individual), with aggregated results published.

Up front: I’m not in a position to drive this. I am writing this at 23:00+, after reading [RFC] Forbid AI-generated communication

Draft questions:

About you

  1. Your role: listed in a Maintainers.md file / active reviewer not listed / other
  2. Area(s): Clang, LLVM IR/opt, a backend, lld, LLDB, MLIR, runtimes, other (coarse on purpose; can decline)
  3. Is reviewing part of your paid job? Mostly / partly / mostly personal time

Load

  1. Hours per week on review, roughly: <2 / 2-5 / 5-10 / 10-20 / 20+
  2. Compared to a year ago, your review load is: much lower / lower / same / higher / much higher
  3. How often do you skip or abandon a review because of the effort it would take? Never / rarely / sometimes / often
  4. How often do PRs in your area get approved or merged by others when you would have asked for changes? Never / rarely / sometimes / often

Sources

    1. Which contribute most to your load? (pick up to 3) Low-quality or AI-generated PRs / overall PR volume / design disputes / CI and flaky tests / notification noise / post-commit breakage / too few other reviewers in my area / other
  • 9a. Of the AI-assisted PRs you reviewed in the past six months, how often did the author clearly understand the change and need no more help than usual? Never / rarely / sometimes / often / can’t tell
  • 9b. How often did you end up doing most of the work to make an AI-assisted PR landable? Never / rarely / sometimes / often / can’t tell

How you’re doing

    1. Overall, how would you describe your level of burnout from LLVM work? (1 = none, I enjoy it; 5 = completely burned out)
    1. In the past six months, have you considered stepping down or reducing your involvement? Yes / no / prefer not to say
  • 11a. Compared to a year ago, how much do you enjoy reviewing? Much less / less / same / more / much more

What would help

    1. Rank what would help most, or pick “none of these”: stricter AI policy / limits or vouching for new contributors / stricter commit access / more reviewers in my area / better tooling and automation / clearer permission to ignore or close PRs / funding or employer support / none of these
    1. Anything else? (free text)

AI disclosure: the text and the draft questions came out of a conversation with Claude. My prompts, roughly:

[RFC] Forbid AI-generated communication , especially a few posts I used to create [RFC] Forbid AI-generated communication - #19 by MaskRay “Regarding Aaron’s point about maintainers considering leaving, what else can we do?”

14 Likes
2

CC @project-council

3

I think this question needs to be flushed out. I have seen somewhat of a bimodal distribution here. Where there are folks using LLMs to help but are doing most of the work themselves and then there are folks using LLMs and the reviewers have to do most of the work on the PR.

So this question I think needs to be two questions to fully capture what is going on. One set of folks are not a problem while the other set are causing the increases in workload and burnout.

1 Like
4

I worry this might not capture cases like “my job requires me to do so, but I loathe having to deal with the slop and so my job is now less fun”. Presumably this would be captured by the scores to other questions indicating the workload is higher etc, but I would caution against taking lots of “No” answers to mean that there isn’t a problem, i.e. you would need to analyse them in combination with other answers to understand them, if not adding a fourth option here to capture that kind of case.

1 Like
5

We will discuss this at the project council, but I would highly suggest having someone qualified at least phrase the questions and, ideally, help us interpret the results. It is very easy to make questions leading or open to interpretation in the wrong way even for folks who don’t have an opinion on the subject.

3 Likes
6

Q9 splits into 2. Q11 splits into 2. Edited the post. Feel free to take the draft questions by whoever phrases the final version.

  • 9a. Of the AI-assisted PRs you reviewed in the past six months, how often did the author clearly understand the change and need no more help than usual? Never / rarely / sometimes / often / can’t tell
  • 9b. How often did you end up doing most of the work to make an AI-assisted PR landable? Never / rarely / sometimes / often / can’t tell
    1. In the past six months, have you considered stepping down or reducing your involvement? Yes / no / prefer not to say
  • 11a. Compared to a year ago, how much do you enjoy reviewing? Much less / less / same / more / much more