As a contributor who frequently follows project updates, when reviewing pull requests and issues, I often find that the details section of submitted pull requests includes trivial everyday details. This makes it difficult for administrators to categorize these pull requests or issues.
Therefore, I suggest adding a keyword recognition feature to GitHub Actions for pull request comments. If certain keywords, such as <code block symbol>, test, solution, etc., are not recognized, GitHub Actions will add a tag indicating that “key information still needs to be supplemented.” Once the pull request submitter has supplemented this key information, the request can proceed to the review process normally.
The tone of the GitHub Actions bot could be made slightly more tactful to make those who contribute to open source feel respected.
Do you mean that the PR is missing (i.e., omits, not “includes”) those details?
This proposal has a similar dynamic to overly prescriptive PR-description templates. I have also found the recent rise of lengthy PR descriptions to be emblematic of LLM-generated noise.
There is no reason to block the review process merely because the test strategy is not described in the PR description. For example, a small change might add a single test file whose header comment already explains the testing approach.
More generally, we should avoid systematically encouraging contributors to add information that is wholly redundant with the code, tests, or other materials already under review. This is not to say that “completeness” of the PR description is not important, but an automated checklist is a rather blunt tool for a task that often requires human judgment.
Thank you for your suggestion! the following is my point of view, if you have time, you can use it as a reference, please feel free to correct what needs to be improved.
Indeed, what you said makes a very good point! Moreover, I’m not sure if there are any tools in GitHub Actions that can perform static analysis of code formatting (I’m referring to comments). Natural language itself is a process that requires human definition, and my idea of hindering the review process because the pull request description wasn’t well written is too one-size-fits-all.
My initial thought process focused solely on two factors: incomplete pull request descriptions and the use of automated tools to generate descriptions that had little relevance to the submitted code.
Understand, this original solution of mine cause the user to focus on the template to write well, okay? Can you pass the trial? In addition, I deviate from the real topic when submitting the information, which I really did not consider.
So could we design a set of GitHub Actions suggestions based on the official guidelines to guide users in writing concise and easy-to-understand descriptive information, while also facilitating maintainers’ writing? (This is merely to guide users.) Or could we inform new maintainers in the initial welcome message within GitHub Actions, following the standards you just mentioned? I believe providing appropriate guidance is still very necessary.
In addition to this, I have noticed a detail about the project submission information, if you are interested, you can take a look:
Additionally, I’ve noticed that when merging commits in the llvm-project’s GitHub actions, the merged commit logs often only include a description and a quick link to the corresponding pull request discussion.
Imagine someone with write access who isn’t entirely clear on what they’re doing, and the unit tests haven’t detected a hidden issue. They see the automated checks have passed and click merge. Then, for some unknown reason, a bug is triggered under certain extreme conditions. When maintainers try to investigate using the commit logs, it will be very difficult to pinpoint the exact line of code causing the problem and how to quickly reproduce and roll back the issue (although this is rare).
Therefore, even for small changes described in comments, I still recommend including a description in the pull request description explaining the API’s purpose and the unit test commands to facilitate future backtracking.
We cannot come up with any set of heuristics that will cover every case without having false positives/negatives firing all the time. LLVM is too diverse a project with too many different areas to be able to codify a template/set of templates or actions that flag things up in a PR description. We don’t really even have a rigid guide in the docs on what should be in the PR description. Consider a PR that updates the documentation only due to a missing command-line option in a tool’s command guide; the set of information needed for this may vary wildly for a complex PR that e.g. implements a new pass in the backend.
PR descriptions are supposed to be reviewed as part of the PR process. In this way, we can judge and feed back whether they appropriately articulate what they should. I’ve on more than one occasion pointed out the unnecessary verbosity of PR descriptions and asked for them to be condensed down. It’s also worth noting that we have banned generating PR descriptions being generated by LLMs, so people posting such PRs are in violation of our documented AI Policy.
If they aren’t entire clear on what they’re doing, then presumably they got write access because they are familiar with a different part of the LLVM codebase to the one they are working in. Such a developer should know that the LLVM policy is that no-review commits are generally only allowed for areas that a developer is familiar with and/or for clearly correct changes like typo fixes. As such the statement “They see the automated checks have passed and click merge” should never happen. If this did happen, the change would be ripe for a revert and possible further action taken against the contributor, depending on how serious the incident was.
I’m sorry, I don’t follow how “hard to spot issue from the commit messages” is solved by “more PR description including the unit test commands”.
The heavy legacy system is indeed a major challenge. Even when I’m navigating between different modules, offering advice and oversight, I’ve noticed the sheer number of sub-modules. I still need time to understand and learn how to use the interfaces in the official source code. Additionally, when debugging locally, I occasionally encounter branch conflicts due to not having memorized basic commands. Sometimes, because the formatting tool version is too new, pull requests show hundreds of lines of changes, disturbing many maintainers.
Furthermore, I personally hold a neutral but not entirely welcoming view of LLM tools. Since most of the interfaces provided by LLM are outdated, I mostly rely on the official documentation. If I can’t find it immediately, I refer to how others use the interface in the source code. To quote what I said before when reviewing code in the community, the answers provided by LLM tools are only for reference and may lead to illusions. Also, when reviewing code, I occasionally find that LLM tools generate pull requests with similar descriptive features.
Additionally, I started working on this project in February, helping others review their code. I’ve gotten a general idea of ​​what’s going on, but when I actually started writing code myself, I realized how deep it was. Sometimes unit tests would take almost a week to complete. I also used LLM tools for assistance, but only for reference; I didn’t use the source code. I still had to check the correctness of the LLM source code against the context of the source files. I’ve been thinking about how to optimize the workflow, and it wasn’t until you pointed me in the right direction that I realized the reality is far from ideal.
Indeed, the actual implementation is extremely difficult! And the llvm-project is already quite mature, so I will follow your advice and spend some time carefully studying the official documentation and community guidelines. Once I’ve developed a low-cost implementation solution based on the official documentation, I’ll come back to you for evaluation. Finally, thank you for carefully evaluating my suggestions and for pointing out many potential issues I hadn’t considered.
I do sympathize, I also feel like due to mainly LLMs the summaries are much larger but often miss key information at the same time. We are seeing something similar with issues as well. I don’t think a prescriptive solution is the right way to go, it just creates more work for everyone else and won’t ever be complete.
I think at least for now, the solution is to explain what we expect, pointing to any documentation we have. Depending on how overly verbose the summary is maybe some suggestions on how to trim it down to something more digestible. Most contributors will get it and will improve with feedback but if not then that can be escalated.
You might want to create a template for yourself if you find that you are making similar comments regularly.
Perhaps the line you draw for “generated by LLM” is different because I do not read the policy as making such a prohibition. The documented policy merely “strongly recommend[s]” the practice of writing PR descriptions without LLM assistance. A human-reviewed LLM-assisted PR description that started with material generated by an LLM still satisfies the letter of the policy.
FWIW, I wish the policy required more substantial human authorship for PR descriptions.
You’re right, I misremembered the exact wording. I do agree with you that I think we should outright ban generation of PR descriptions by AI tools, with the possible exception of specific bulk-generated changes that get pre-approval beforehand (this of course does not include banning editing/translation by LLMs). That’s a separate discussion that would need its own thread though.
I agree with your point! My proposal indeed failed to consider the fact that the project’s massive scale makes it difficult to reach a consensus.
I understand. If that’s how you put it, then I feel my proposal itself is somewhat unreasonable. Just as this comment says…
If it’s just a minor change, like a typo in the official documentation, then there’s really no need to write a lengthy description. While I did suggest using GitHub Actions to guide users, I honestly didn’t consider the user’s perspective at the time. Too many prompts might actually dampen people’s enthusiasm for contributing.
Although my initial intention was to improve the efficiency of reviewers, after careful consideration, I realized that having contributors fill out templates to improve review efficiency is indeed a false proposition.
Speaking as an individual and not in the capacity as an area team lead, I don’t think there is sufficient motivation here to want to create heavy rules about how we annotate commits.
Personally, I have had several negative experiences with fussy project rules about PR description formatting. They’re the kind of paperwork that people tend to ask agents to fill out on their behalf. I’d want to hear a really compelling reason that would create value for the community to outweigh the cost of adopting new commit tagging policy.
To briefly explain, my thinking focuses on protecting the limited energy for real-do-true-things-reviewers.
Even from an engineering design perspective, while it’s true that it will worsen the user experience about the beginners, it’s still necessary for novice maintainers to learn some basic theoretical knowledge. Otherwise, the number of low-quality merge requests that get mixed in later will only increase.
My initial approach is to add a message to the github actions bot about greeting. Suggesting set pre-submission check script already present in the official documentation be implemented, along with a link to the official documentation.
This would conservatively filter out about 10% of those trying to scramble for submissions, forcing them to stop and seriously consider their options.
Additionally, I read the official documentation and learned that the official documentation already includes a very comprehensive pre-commit check script located in the repository’s llvm/utils/git/ directory. Contributors only need to install the Python environment and create a symbolic link to use it directly, with almost zero time cost and saving a lot of time spent looking up commands.