[RFC] Forbid AI-generated communication

[nikic]

LLVM’s AI tool policy is fairly liberal and allows essentially all uses of AI, as long as there is a “human in the loop”, who takes responsibility for the contribution.

Now that we have more experience with AI-assisted contributions, I would like propose a change to the policy, which is aimed at rejecting extractive uses more thoroughly, while retaining the beneficial ones.

In particular, this RFC proposes to ban the use of AI-generated or substantially AI-assisted textual communication, including, but not limited to, for pull request descriptions and RFC texts. It does not make any changes to the allowable uses of AI for code contributions.

Motivation

The main motivation for this is the very large amount of low-quality pull request descriptions we receive, especially from new contributors. These are often extremely verbose, contain huge amounts of unnecessary information (such as about routine testing), as well as gratuitous formatting.

Such pull request description are a drain on our limited reviewer bandwidth, both because they take more time to read in the first place, but also require substantial massaging to bring them into a mergeable form.

Additionally, AI-generated pull request descriptions deprive the contributor of the opportunity to understand what a PR does by themselves, and express it in their own words. This is an important sanity check for the “human in the loop” contribution model.

Of course, this issue is not limited to just PR descriptions, even if they are the most pressing concern due to the sheer quantity of incoming PRs. We’ve also had issues with overly verbose and hard to understand AI-generated RFCs, etc.

Proposal

The proposed policy would be something along these lines (in spirit, not necessarily in wording).

Any AI-generated or substantially AI-assisted textual communication must be clearly labelled as such, and enclosed either in a blockquote or a <details> section.

As this is not suitable for pull request descriptions (which become commit messages), they must not contain any AI-generated text. If you want to post additional AI-generated information, do so in a separate comment (with appropriate labeling and enclosure).

The same applies for RFCs, which must be fully understandable without reading AI-generated content. However, such content (with appropriate labeling and enclosure) can provide non-essential auxiliary information, such as data analysis.

“Substantially AI-assisted” includes any case where the description is initially generated by AI and then edited by humans. It does not include the use of AI for translating an initially human-written text and does not include the use of AI for proofreading.

“Textual communication” includes, but is not limited to, pull request descriptions, issue descriptions, comments on PRs and issues, RFC texts and comments on Discourse. It does not include (at present) code comments.

Examples of appropriate labelling and enclosure are given in the following.

Variant 1:

I asked Claude to review this PR, and I think it makes a good point:

AI generated text goes here.

Variant 2:

Some additional data analysis performed by Codex:

AI generated text goes here.

Notes

This policy will have collateral damage. There are people whose communication, while being very clearly AI-assisted, is still concise and useful. However, there is a limit to how nuanced we can make policies without rendering them useless.

For example, the fact that our current AI tool policy “strongly encourages” writing PR descriptions yourself turned out to be essentially useless. AI-generated descriptions are not explicitly forbidden, so that’s what we get.

At this point, I believe that the damage done by AI-generated PR descriptions is much higher than the damage that will be done by forbidding them.

Finally, I also don’t want to start any witch hunts about AI-generated text. The problematic cases tend to be exceedingly obvious. This is a question of policy enforcement: The intent is to be able to easily reject egrogious cases, not to hunt down any text that might possibly smell of AI involvement. Your use of en-dashes is fine.

[mehdi_amini]

@nikic

Any AI-generated or substantially AI-assisted textual communication must be clearly labelled as such, and enclosed either in a blockquote or a <details> section.

Okay, that’s not an outright ban. Seems reasonable to me!

@nikic

As this is not suitable for pull request descriptions (which become commit messages), they must not contain any AI-generated text

That feels completely overly restrictive to me: AI generates perfectly fine description when correctly instructed to. Unsteered there is a lot of variation: Claude in particular used to be terrible and very verbose (I think recent version got better, not sure), it’s pretty easy to tell when reading a PR description, however you can steer it drastically: “Provide a high-level description of the change in 8 lines max, include context about fixing issue #12345 and mention follow-ups about XYZ”, I general review the output and may iterate 2/3 times till I get something “right”. The end result is much better than what I would do in general.
Same goes for RFCs of course.

@nikic

This is a question of policy enforcement: The intent is to be able to easily reject egrogious cases, not to hunt down any text that might possibly smell of AI involvement.

Every other day, I have to comment on a PR asking the author to trim down their description significantly because it looks like “AI slop” (And ask them to disclose because most of the time it’s not). I had someone pushback on fixing their description once, so, while slightly annoying to have to be the “bad guy” telling people about it, I feel totally the opposite: the policy you propose will have significantly more drawbacks than benefits.

That’s not even mentioning the enforcement aspect: we’ll end up with arguments about whether a description is AI-based or not, instead of whether it fits guidelines of “how to write a good commit message”.
For example, in terms of a good commit message, I am often annoyed about unrelated things in the PR description (author asking questions about the review and things like that) because the author doesn’t realize the PR description will become the commit message and they use the PR description as a way to prompt the review discussion (including @name tagging) with things that really don’t belong to the commit history.

[jrtc27]

Do you have an alternative proposal that would still prevent all slop whilst allowing the cases you deem acceptable?

Bearing in mind that the worst that comes of this is “person who uses and reviews the output of heavily-hinted LLMs has to go back to writing a few paragraphs of text by hand”, I would much rather err on the side of having clear anti-slop messaging that requires marginally more effort from such people than try to carefully carve out exceptions and see them inevitably be abused and argued as to where the boundary is.

[ilinpv]

I don’t think it should matter too much how the wording was produced, whether with AI assistance, advice from a colleague, or by other means. What matters more is that the author understands the content, stands behind it, and that it meets LLVM’s standards for good contributions.
Rather than trying to police how communication is produced, I would prefer us to focus on clearer and more explicit guidance for what a good PR description and commit message should contain, and what should be avoided, regardless of how the author arrived at the final wording.
PS For transparency, this comment itself was drafted with AI assistance under my guidance, and I reviewed and stand behind the final wording.

[vtjnash]

@nikic

It does not include the use of AI for translating an initially human-written text and does not include the use of AI for proofreading.

I think there’s a slight tension here between the earlier text “no AI-generated text” and this acceptable use to proofread, so I think the final draft needs to try to be clear that the policy isn’t about forbidding AI-generated text, but about not posting content which is not respectful of the reviewer time and attention – I think spell/grammar/consistency-checking my replies is more respectful, when the intent isn’t being machine-generated.

As another data-point, the Julia community stewards are considering simplifying the “except for AI translation” clause by changing the guidance to prefer writing in your native language, and assume the audience also has access to the same AI auto-translation tools, so it may be unnecessary for the author to use them in advance:

Clarify AI guidance, add dedicated AI flag (#20)

This is still a bit of a new experiment, having just enabled auto-translation for julia’s discourse forum and begun testing out the edge cases of it:

Testing automatic translations to English

[shafik]

I believe we need something stronger and broader than this but this is a good start. We definitely have a number of maintainers that are on the verge or who are burnt out due to the load of LLM generated PRs that are just plain extractive. It is additionally problematic in that these same contributors pump out a large number of PRs wrt to other contributors. The ideal is that you start out using LLM tools but after a few PRs you improve and the quality goes up and the author does more and more of the work themselves. OTOH what we are seeing is that the quality does not improve and the maintainers end up having to hand hold on almost all of their contributions. This is just exhausting.

I have to say separately this is becoming a real problem for triaging as well. I just have not had the time to think about it more thoroughly or come up with a post on it.

[erichkeane]

@shafik

I believe we need something stronger and broader than this but this is a good start. We definitely have a number of maintainers that are on the verge or who are burnt out due to the load of LLM generated PRs that are just plain extractive. It is additionally problematic in that these same contributors pump out a large number of PRs wrt to other contributors. The ideal is that you start out using LLM tools but after a few PRs you improve and the quality goes up and the author does more and more of the work themselves. OTOH what we are seeing is that the quality does not improve and the maintainers end up having to hand hold on almost all of their contributions. This is just exhausting.

This is absolutely true. I’ve been working hard with a handful of extractive-contributors with the hope they would get better, and even after dozens of commits, they are just as (if not more so) extractive as before.

[shafik]

If we are to argue that “there must be a human in the loop”, “author and is fully accountable for their contributions” and " should be able to answer questions about their work during review" then the low bar here is that they should be able to in their own words summarize: What, Why and How. What did you change, it does not have to exhaustive but should provide enough details so a reviewer(s) knows what to expect. Why did you make the change, understanding the motivation for the change can often clarify choices you made that without having a rationale could be surprising. Finally, how did you make the changes, did you use a different algorithm or data structure, did you add extra checks or assertions, what design choices/tradeoffs did you make etc I am quoting my own post on how to write a good summary here in case anyone wants to read more on that, b/c everything in that post is relevant to this discussion.

If they are not able to at minimum do this, then they should not be submitting the PR b/c they don’t actually understand what the PR does. So I think this proposal is both reasonable and should be the lowest bar we are willing to accept.

[AaronBallman]

I appreciate that we’re continuing to try to tighten our AI policy wording, so I’m in support of the intent here.

However, I’ve been frustrated enough with the LLM slop that I continue to believe the most defensible policy is a “no use of LLMs” policy. I know that’s not a popular opinion, but my experience under our existing policies has been wholly negative, to the point where I’m silently giving up on reviews when I start to smell an LLM being heavily used in them. As someone in the “review everything nobody else gets to role”, it is absolutely untenable and it has made me reconsider whether I want to continue to be lead maintainer for Clang on several occasions because I do not feel capable of meeting that responsibility given the current policies in place.

I know I am not alone; just this morning I had a long-time Clang maintainer privately tell me how close they are to stepping down as a maintainer or leaving the project entirely over this exact problem. We are putting a lot of faith in the idea that this policy somehow helps us grow the community, but when multiple long-time maintainers are considering leaving the project, I think the burden of proof needs to shift to the people pushing for LLM use that this is actually adding value instead of extracting it.

As you said:

@nikic

Finally, I also don’t want to start any witch hunts about AI-generated text. The problematic cases tend to be exceedingly obvious. This is a question of policy enforcement: The intent is to be able to easily reject egrogious cases, not to hunt down any text that might possibly smell of AI involvement. Your use of en-dashes is fine.

It’s these exceedingly obvious cases that I’m talking about wanting a ban for. I don’t care if someone used an LLM to help them come up with some extra test coverage or to help them find other places they need to update in their PR. But when I get 20-30 PRs opened in a short period of time from the same (new) contributor, all in different parts of the compiler toolchain, it’s abundantly clear that I’m not helping a new contributor onboard, I’m just helping them figure out their prompts and the beatings will continue forever.

I want to +1 this RFC because it’s at least adding more restrictions, but honestly, I don’t think the proposed changes meaningfully help. The problem isn’t technical, it’s social: this adds more rules for the community to follow but does not add any teeth to the policy for handling problems. It’s performative, just like our “please review and acknowledge the AI policy” comments on PRs.

[tstellar]

@AaronBallman

I know I am not alone; just this morning I had a long-time Clang maintainer privately tell me how close they are to stepping down as a maintainer or leaving the project entirely over this exact problem. We are putting a lot of faith in the idea that this policy somehow helps us grow the community, but when multiple long-time maintainers are considering leaving the project, I think the burden of proof needs to shift to the people pushing for LLM use that this is actually adding value instead of extracting it.

In a lot of ways, our policies don’t really matter, because reviewers can just ignore patches they think are extractive, even if the patch complies with our AI policy. We can’t force reviewers to review a PR.

On the flip side, though, if we have a “no use of LLMs” policy, this doesn’t actually stop people from submitting AI slop PRs, and we still end up with the patches in the queue and someone still has to review them even if it’s just to determine if they meet our AI policy.

In the past, we’ve added requirements for obtaining commit access, because commit access can potentially be destructive, and we need to be able to trust people. We are now at the point were submitting extractive AI PRs can be considered destructive so I wonder if we have to do more to restrict new contributors from submitting PRs.

[AaronBallman]

@tstellar

On the flip side, though, if we have a “no use of LLMs” policy, this doesn’t actually stop people from submitting AI slop PRs, and we still end up with the patches in the queue

Fair points, but pushing back a bit: that’s true of all of our developer policies, to varying degrees. We can’t stop people from submitting incorrectly formatted code, or comments which violate the CoC, etc. “We can’t stop people” isn’t super compelling to me because we can stop people, but we need the community to feel empowered to do so. However, I think “is this using an LLM?” is always going to be difficult for anyone to feel comfortable answering by themselves, similar to “is this violating the CoC?” and so I definitely agree with:

@tstellar

and someone still has to review them even if it’s just to determine if they meet our AI policy.

I don’t think we want an “LLM committee” like we have a “code of conduct committee” due to scaling (there is far more LLM use than CoC violations, thankfully!). But having a CoC at all is a signal “we don’t do that kind of behavior here” which stops some people from coming through the door, and I think the same can be said of a ban on LLM use. Perhaps instead of an LLM committee, we decide that if at least two reviewers think something is from an LLM, it gets closed as a policy violation and it’s on the author to reopen & refute it if they want? This would still rely on a ban on LLM use; the more subjective we make the policy, the less likely it is people will feel comfortable labeling it (we already don’t get much use of extractive due to it being subjective).

A perhaps slightly different idea along the lines of restricting PR submissions: you can only have one PR open at a time using an LLM (perhaps maintainers are allowed to have more though?). If we switched AI attribution to use an explicit label that anyone can add to the PR or issue, then it might be feasible to automatically enforce this. When the label gets added, a bot checks for other open PRs from the same author which also have the label on it, if other PRs are found, it leaves a comment about violating policy and reviewers can decide whether to close the PR based on that automatic comment… but they can also “dismiss” the comment and provide review if they’d like. (But new contributors can’t add labels to their own PRs which kind of makes this frustrating too, but maybe that restriction will change in the future?)

[Felipe]

@AaronBallman

I appreciate that we’re continuing to try to tighten our AI policy wording, so I’m in support of the intent here.

However, I’ve been frustrated enough with the LLM slop that I continue to believe the most defensible policy is a “no use of LLMs” policy.

I’m in complete agreement here. To be totally honest, I have stopped engaging with most LLDB PRs where the description is obviously AI slop. There have been too many instances in those PRs where the responses to my comments were also slop, and that just makes me lose any will to engage further.

[avogelsgesang]

@AaronBallman

A perhaps slightly different idea along the lines of restricting PR submissions: you can only have one PR open at a time using an LLM (perhaps maintainers are allowed to have more though?).

I wonder if this should be generalized from “one PR using an LLM” to “one PR for all people without committer rights” independent of AI use.

For new people onboarding to the project, it would likely be a good idea anyway to go one PR at a time, so they can incorporate the feedback they received on the first PR into their second PR before even opening it.

The bot could reply with a helpful message, guiding new people towards a “crawl, walk, run” approach. I personally consider that to be a best practice for onboarding to any open-source project, anyway. And it would have the additional benefit that no judgement call re “did this PR use AI?” is needed.

[oscardssmith]

The counterpoint here is that LLVM policies of requiring test-precommit and single commit PRs can make it difficult to have a single PR, even if you only have 1 PR that you care about. It can be useful or necessary to have multiple PRs to provide justification for each other.

[mehdi_amini]

@jrtc27

Do you have an alternative proposal that would still prevent all slop whilst allowing the cases you deem acceptable?

Yes, and I mentioned it:

@mehdi_amini

we’ll end up with arguments about whether a description is AI-based or not, instead of whether it fits guidelines of “how to write a good commit message”.

We should have a guideline about what a good commit message should be (probably also examples of “bad PR descriptions”), whether I typed it from scratch or got the help of an agent shouldn’t matter, only the result count.
[RFC] Forbid AI-generated communication - #4 by ilinpv explained it pretty well I think.

As an anecdote, when I had one new contributor who pushed back when I mentioned that their description looks overly AI-generated, they insisted they wrote it by hand.
I copy-pasted their PR description into ChatGPT, who told me that 75% of it looks like AI-generated with obvious tell. What I told the new contributor though is that it doesn’t matter that they didn’t use AI here, the fact that it “looks like AI slop” os what makes it a bad description, and I provided a link to some guidelines for writing good commit messages.
They fixed their PR description: whether they used AI for it or not, I don’t really care, I was happy with the result.

@tstellar

On the flip side, though, if we have a “no use of LLMs” policy, this doesn’t actually stop people from submitting AI slop PRs, and we still end up with the patches in the queue and someone still has to review them even if it’s just to determine if they meet our AI policy.

Exactly!

This is why I also prefer being able to point a contributor to the guideline that says “Here is how to write a good description” instead of “I suspect you used a LLM here, and even if your PR description is pretty good as-is, it’s just forbidden here on a matter of principle”.

[lgritz]

I’m just a downstream onlooker here, and I don’t really take a stand about the specific policy LLVM authors wish to choose for themselves, but I wonder if it would help alleviate the situation for reviewers if the project had a root level AGENTS.md checked in that would give instructions that these extractive contributors’ LLM tools would follow. It could just give exposition about what is a good commit message and what is clutter that should be left out. Or on the far end of the spectrum, it could be a hard-core “Only human-written commit messages are allowed in this project, agents should refuse to commit or write the message.” Or anywhere in between. I’m not saying that individual problem users couldn’t evade the instructions, but maybe it would shift the average LLM-aided patch toward being less extractive?

[ayzhao]

I’m just a downstream onlooker here, and I don’t really take a stand about the specific policy LLVM authors wish to choose for themselves, but I wonder if it would help alleviate the situation for reviewers if the project had a root level AGENTS.md checked in that would give instructions that these extractive contributors’ LLM tools would follow. It could just give exposition about what is a good commit message and what is clutter that should be left out. Or on the far end of the spectrum, it could be a hard-core “Only human-written commit messages are allowed in this project, agents should refuse to commit or write the message.” Or anywhere in between. I’m not saying that individual problem users couldn’t evade the instructions, but maybe it would shift the average LLM-aided patch toward being less extractive?

This has been discussed before.

[Endill]

@tstellar

In a lot of ways, our policies don’t really matter, because reviewers can just ignore patches they think are extractive, even if the patch complies with our AI policy. We can’t force reviewers to review a PR.

This is true for reviewers, but note that Aaron was talking about maintainers, and they “ensure that commits receive high-quality review, either by the maintainer or by someone else”. I don’t see how ignoring PRs is compatible with that. Even worse, when reviewers who are not maintainers decide to ignore an LLM-generated PR, maintainer’s life becomes even harder. I think we need our policies to accommodate for the behavior you suggest, otherwise the situation encourages maintainers to ignore our policies, which is not good for the health of the project.

[MaskRay]

I share Aaron’s frustration. lld/ELF, MC, and the binary utilities have received a steady stream of AI slop PRs over the past few months. They come from first-time contributors, from people who shed the “First-time contributor” banner after one heavily-mentored merge, and from some who got commit access under the bar I questioned in Revisiting commit-access criteria (2026)

A system without proof of work is easy to spam. The effort it took to write a patch used to be a commitment signal. LLMs removed it, and the cost shifted from the contributor to the maintainer. A hand-written PR description restores a small piece of that signal, so I support the direction here.

With the same tools, experienced contributors produce patches of much higher quality. LLMs amplify judgment rather than replace it, and judgment is only visible through track record. So in practice, the GitHub author banner has become my prior. “First-time contributor” now means low quality in nearly every case I see, and “Contributor” has depreciated significantly this year. Breadth is another signal: a “Contributor” sending substantive changes to unrelated areas within a short time is usually iterating on prompts rather than learning the codebase. (The same mechanical cleanup applied across many libraries is a different pattern.) These signals set my review order and how much time I’m willing to invest up front. That’s unfair to the occasional genuine newcomer, but my bandwidth leaves no other choice.

Part of this is the platform. Self-hosted review (Phabricator, mailing lists) had registration and tooling friction that acted as a mild proof of work. I noted in 2023 that GitHub is more contributor-friendly but less reviewer-friendly. With LLMs, that trade-off has turned against us, and llvm-project’s visibility makes it an attractive target for contribution farming.

---

A few things I’d add.

First, “mentor, don’t rewrite” used to be a good rule, but it barely works today. When a patch needs more rework than review, a clean replacement (“Supersedes #NNN”) is better for the project, and we shouldn’t feel guilty about it. Credit the original author if you used their code or diagnosis; otherwise the cross-link is enough. A side effect: the replaced PR doesn’t count toward the author’s status (“First-time contributor” → “Contributor”) or the merged-PR threshold for commit access.

Second, I can’t simply ignore slop in areas I care about most, because another committer may approve it. An approved but problematic PR can be merged within two days (and I really cannot catch every email notification), sometimes by yet another committer. This has happened a few times in my areas. Rules for authors only cover half of the problem.

Third, on collateral damage. As rnk noted in the cl::opt thread, some tech debt paydown would never make it to the top of the priority heap without AI tools. The text of [RFC] Declare library command line options in TableGen, one struct per library and a few foundational patches are heavily LLM-generated, so they would be non-compliant under this proposal. (I did spend significant time on them, but less rigor per patch than I’d normally apply. I’m not proud of that, and I’m grateful to my reviewers.)

Finally, I’d like us to try something more drastic as a time-boxed experiment: close the PR path to unvouched accounts, as Ghostty does with Vouch. A vouch list can be spammed too, but it’s a system we control, unlike GitHub, where obvious improvements we’ve wanted for years never arrive. Even one month would tell us whether reviewers feel the load drop. If that’s too drastic, the per-author open-PR limit from the contributor policy refresh (RFC: Contributor policy refresh), with a limit of 2-3 rather than 1, is a smaller step.
People who want to contribute would still have a path. An issue with a good diagnosis or reproducer is a real contribution, and a maintainer can land the fix. As rnk put it: “These days, I push a commit to my fork and say ‘here’s the things this needs, feel free to merge to your pr or rewrite to your satisfaction’”. I’d also like us to care less about whether a first contribution is recorded as Author. A Co-authored-by trailer or a credit line in the commit message recognizes the work; the GitHub badge isn’t the point.

[Pierre-vh]

@AaronBallman

However, I’ve been frustrated enough with the LLM slop that I continue to believe the most defensible policy is a “no use of LLMs” policy.

I agree, at least for technical documentation/descriptions.

There is something uniquely draining about reading LLM-written text (especially Claude’s). It’s uniquely verbose and dances around the point. I feel like time passes twice as slow when I read a wall of LLM-generated text, and feel twice as tired after.

I also really don’t buy that it’s a problem of “learning how to direct the LLM to give better output”. That’s purely a matter of taste. I personally dislike the overly catchy prose of AI and no amount of shortening it helps. Another contributor might be fine with the prose but hates the volume. There is no objective criteria to enforce because different people spot different traits.

On a more fundamental level, I think asking people to write their documentation (including PR descriptions) themselves is fair because it lets the author double-check their knowledge by putting it “on paper”. It also helps reviewers. If I see a human written text, leaps of logic, misunderstandings can become apparent quickly. I think LLMs will generally hide leaps of logic behind an authoritative, catchy tone, making basic issues (such as “why didn’t we try this alternative”) hard to spot.

I stopped counting how many times I learned things by writing down documentation, explaining my logic in my own words, and getting corrected. It really works, at least for me.