There’s a lot to think about in this thread and it’s been on my mind quite a lot since it started. I have the following thoughts:
-
The subset of the linked Fedora policy covering contributions seems like a good starting point. For Reid’s PR I have a concern about the level of indirection: defining “extractive” contributions and a general policy on expected quality and then trying to have the AI contributor policy fall out of that is a noble effort, but it feels like it’s trying to take on a much bigger task when a more targeted policy on tool/LLM-assisted contributions could be easier to agree. Maybe it ends up being shorter, maybe not, but I hope it would be easier for contributors to relate to their own situation and easier for us to agree as a community than something broader in scope.
-
Not everyone agrees, but the policy we previously adopted and the variants most discussion have focused on allow for the use of LLMs as an aid in preparing a patch. The issue, as I see it, comes if that usage breaks our human-driven review system. I would be very unhappy and uncomfortable with a policy that controls the tools that contributors can use except as some kind of last resort. I believe attempting to refine our current policy is the sensible next step, ensuring things like people take personal individual responsibility for the work they submit and they commit to engage directly in a bidirectional review process. If someone has taken time to study code, stands by it as if it were their own, and will personally engage in the back and forth review process then I don’t mind if they wrote it from scratch, used sed, or had a room of monkeys with typewriters write a billion drafts until one worked.
-
Some ideas of behaviours we could either explicitly call out or ensure are covered by some more general wording:
- Submitting a PR where you either don’t understand and stand by the changes, or at least don’t flag which bits you are personally unsure of. This wastes reviewer time, and LLMs being confidently wrong at times can mean that reviewers go on a wild goose chase trying to understand an assertion made in a submitted PR that a human hasn’t actually thought through.
- It is never appropriate to engage in review discussion by taking questions from a reviewer, pasting it into an LLM, and pasting the response back. I don’t know if we’ve seen ths happen in our community, but a colleague mentioned the frustration of going through this loop in a project and I can only imagine the frustration.
- Based on comments in this thread, it sounds like we want guidance that fishing for potential patches through LLM “audits” is unlikely to result in high quality submissions and is strongly discouraged, especially by new contributors. (Established contributors are better placed to make a judgement call on whether something is worth submitting or not, so I’m not sure I’d rule it out entirely for someone who knows what they’re doing).
-
When it comes to “disclosure”, I think the Fedora policy strikes a good balance here with “significant assistance”. Given the wide range of ways of using AI tools I think requiring disclosure for any usage is overkill (do I disclose if my search engine gave me a summary answer for my python docs query and I used that rather than clicking through?). I think something like “If your contribution has largely been generated by a tool (e.g. an LLM, custom script, clang-tidy, etc) you should flag this in your pull request as an aid to reviewers.”
-
One thing that’s been bouncing around my head is about how to make more concrete the idea that you understand and “take ownership” of what you submit. The PR description / commit message is a really important part of this, and writing it yourself seems like a potentially useful litmus test. If you find it useful perhaps you use an LLM to aid in translation from your native language, help with phrasing, or to check it that all seems fine. But I think you can still do those things and have a message that is predominantly your own work, and this would be a totally reasonable thing to require as “cost of admission” for as long as our reviews are primarily performed by human contributors.