The roundtable began with someone asking what the current AI policy is. @rnk responded that initially, one company had concerns about labeling, so the foundation had some discussions. There were also concerns about copyright. There was an RFC, and the consensus was that if AI generated copyrighted violating code it was removed. Later on, there are concerns about AI generated code adding burden to maintainers.
@jeremybennett raised a concern about vendors not allowing AI generated code in the source. There are also concerns about ownership of AI code. The problem is that even today vendors would have to verify ownership of code today.
@chandlerc responded that from a copyright standpoint, the copyright status of AI generated code is well-settled. But this is only in a US context
@cmtice asked: what is AI generated code? What if for example a tool answered a question? @rnk said that we don’t have an answer to that question.
@rnk explained that most of his proposed policy derives from Fedora’s, which emphasizes that maintainers should own and understand their code including if it’s AI generated.
@mtrofin raised a question: what if I generate a small patch where I don’t understand the code, but the reviewer OKs it? The small patch wouldn’t be a huge burden. @rnk responded that we don’t care about whether people use emacs, vim, etc. so we should extend this to AI. For example, if someone generates a small python script for monitoring, then this isn’t a waste of time.
@rnk mentioned that he got 2 types of feedback on current AI policy: one group wants a short, concise policy. The other group wants to outright ban LLMs.
Someone asked a question: what changed now to trigger this discussion? Do we have examples of why this is needed? @rnk responded that historically we got value and ROI growing maintainers by mentoring and teaching newcomers. But newcomers going to LLMs defeats this.
Someone asked who would be the authority for enforcing the policy? @rnk mentioned that it’s always been subjective how we determine what is a high quality pull request. @rnk wants to document something that maintainers can point to rather than having to waste time explaining to everyone why their contribution won’t be expected.
Someone asked whether we have a tool to detect AI generated code? Someone responded that the general consensus is that this is not reliable. There was a follow up question that if we have a policy, how do we enforce this? How do we know something is AI generated? People also proposed retroactively applying this policy and back labeling PRs. @rnk said that no draft policy has explicitly banned AI tools. But people have been pushing for an outright ban. @rnk’s main question is would we want to encourage new contributors to use an LLM? He believes the answer is no, as this is counterproductive. One engineer advocated for requiring labeling AI generated code.
@ldionne commented as someone who spends a lot of time reviewing upstream PR, I have no objection to reviewing AI generated code. Some of these PRs have amazing quality even with AI generated code, and there are some productive discussions where he was engaging with the author. However, it is a problem if the author doesn’t engage with the reviewer on the PR. It would be nice to have a tool/policy to protect maintainer time.
@chandlerc commented that having something to protect maintainers time is good. It’s probably a waste of time to retroactively labeling PRs. Protecting time is good. But so much unknown code has been submitted using AI. We cannot put this back into the box. Focusing on healthy maintainer is more constructive.
@rnk reiterated that the purpose of a labeling policy is to help protect maintainer time.
Someone asked whether or not we are seeing new PRs today due to LLMs, or are more people just submitting large volumes of poor quality PRs in general? @rnk was not sure.
Anshul commented that @chandlerc is saying that banning AI is too big of a hammer for this problem. In the review process, there is a self-correcting process. If a user doesn’t know about their patch, then that user would be ignored in the future. We should be thinking about a smaller hammer approach. @rnk also thought this would be a self correcting problem - if you submit bad patches, then your reputation will decrease. But this is enough of a problem that we need to have a policy so we don’t argue this in the future.
@ldionne commented that if a PR is labeled as extractive, it should be closed instead of opened, because an opened PR is something that requires attention. @rnk observed that some people may not be aware that there is recourse if PRs are closed. We should leave the door open - it seems like the polite thing to do. @brenfwd mentioned that alternatively, the PR should be converted to a draft PR instead of closing it, as it would not be in the maintainer’s open review requests and it won’t be closed.
@chandlerc observed that the highest quality generative AI coding tools have recitation checking tools to protect against generating copyrighted code. This is generally better than people copy-pasting code. Also, there are people and tools who continuously check for licensed code in LLVM - for example, Microsoft detected copyrighted code using some tooling.
@rnk started a poll whether there should be a tool-neutral policy protecting maintainers’ time. Everyone raised their hand
Someone asked if before AI, did we have a policy on whether or not a contribution was wasting people’s time? @rnk observed that nuisance contributions have been an issue before. It makes sense to have a policy now with the rise of LLMs.
@rnk also commented that maybe another criteria should be whether or not a contributor can explain their code.