A couple of us (@cardigan1008, @dtcxzyw) have been experimenting with using LLMs to assist in the review process. These efforts have been focused on different areas such as automatic crash/miscompile reproducer generation and programmer’s manual enforcement but overall are in the same vein of using LLMs to assist with reviews and are complementary. We want to propose building out some infrastructure in an upstream LLVM repository to generate these reviews so that we can consolidate our efforts. Our primary motivations here are to provide trivial style/nit fixes to save reviewer bandwidth (e.g., enforce coding standards and area-specific guidelines like the InstCombine contributor guide and the CI best practices page), and to catch critical bugs to improve the code base quality.
NOTE: We are explicitly not proposing automatically putting the generated review comments anywhere where they would be visible to PR authors by default (e.g., as GitHub reviews through an app). They will be visible through a website, but a PR author/reviewer would explicitly need to seek them out. In the future, if we are able to produce high enough quality comments and there is sufficient community interest, we can look into automatically/upon review request posting some of the review comments directly to PRs.
The Proposal
Based on the success of current generation infrastructure (e.g., GitHub - dtcxzyw/llvm-hackme · GitHub and GitHub - cuhk-s3/Archer: Archer: Agentic Code Review for LLVM PRs · GitHub ) and the success of LLM review tooling in other OSS communities like Sashiko ( GitHub - sashiko-dev/sashiko: Agentic review of Linux Kernel code changes · GitHub ), we think that building infrastructure upstream where people can pool their efforts and avoid building out duplicate boilerplate components would be useful for further experimentation in the space. In the past 10 days alone, llvm-hackme has found 21 critical bugs across 78 PRs ( Pull requests · llvm/llvm-project · GitHub ). Archer has found 38 bugs across 200 open PRs and an additional 36 bugs in 328 commits post-commit ( Pull requests · llvm/llvm-project · GitHub , not all issues have been reported in PRs).
We want to start building out said infrastructure in llvm-zorg under a top level folder (e.g., llm-review/ or another name if preferred). This is our main ask. I plan on setting up initial hosting on our (Google’s) end to run this infrastructure and we should be able to provide enough resources (CPU+tokens) for experimentation. It’s also likely we should be able to provide enough resources for full scale usage, but given possible approvals needed, I don’t want to make any hard promises at this time. It will partly depend on resource-usage data from the experimentation phase, but token demand will likely be elastic in that if we run into cost issues we can optimize our usage. In addition to building out the infrastructure in llvm-zorg, we also plan on putting review “skills” in the monorepo under llvm/utils/review_skills. This makes it easier to keep in sync with code/other documentation, and can also help with local development as they are more easily available to IDEs.
Architecture
Fundamentally when reviewing a PR you have a couple elements that you can provide as input including the PR title, description, and diff. You also have one main form of output in the form review comments, which can either have a file/line number attached to be inline, or omit a file/line number to attach to the PR more generally. There are also a couple more common elements the review infrastructure is likely to need such as a build of LLVM with the PR applied for running opt/llc. It’s also important to note that different areas will have different review requirements and allow for different tooling. For example, InstCombine has its own contributor guide ( InstCombine contributor guide — LLVM 23.0.0git documentation ), only middle-end changes/IR only backend changes would benefit from Alive2, and some backends have support for translation validation (e.g., https://web.ist.utl.pt/nuno.lopes/pubs/armtv-oopsla25.pdf, https://youtu.be/znMGXxQC12A?si=3u9Xh_OdyGo75wFH).
Given the diversity in the project, we propose a pluggable architecture that allows for the addition of subagents that can trigger on certain criteria (e.g., a PR touches a certain file path). Subagents would then be responsible for taking in the contents of the PR and generating review comments (e.g., inline comments on style issues, or whole PR comments with validated miscompile/crash reproducers). This also makes it easy for different areas of the project to customize such infrastructure. For example, if I notice a couple of common issues in PRs submitted against llvm-exegesis, it would be easy for me to write up a prompt/subagent to generate comments pointing out such problems and run it on PRs touching llvm-exegesis.
In terms of the actual compute infrastructure, I think running this on a k8s cluster where review jobs get scheduled as separate pods makes the most sense. This would follow the design of the current premerge infrastructure and allow for easy autoscaling. Given in many cases we want to be able to build LLVM and run tools like Alive2, the CPU compute requirements might not be trivial, so autoscaling is important for efficient resource usage. Scheduling jobs as separate pods also allows compute requirements to scale based on what the PR touches and what the requested subagents need. For example, a PR touching a subproject that does purely LLM based review will need little CPU and can be scheduled on a lightweight pool. In addition to a hosted service, we also plan on ensuring that the infrastructure can be run locally to aid in development and also to allow people to get reviews before ever posting a PR.
We plan on having reviews triggered automatically (review is used generically here, this proposal does not include publishing anything to GitHub) through a polling mechanism that looks at new GitHub PRs for select subcomponents. Only people marked as contributors (have landed at least one commit) on LLVM will have reviews automatically requested.. This is for security reasons given we perform an LLVM checkout/build which allows for arbitrary code execution. While this is not a particularly hardened setup, we consider the security risk here acceptable, like premerge. Eventually (i.e., in a separate RFC), we would like to propose enabling contributors to request reviews natively in GitHub using a flow similar to how one requests CoPilot reviews currently.
In terms of foundational dependencies, we plan on using the Google ADK to interface with LLMs. It provides all the features that we need (e.g., tools), is reasonably standard currently, and is not specific to Google’s models. For example, it is trivial to write up ADK to use Claude instead of Gemini as the underlying LLM. We do plan on using Gemini models initially given they are easiest for us to get tokens for, reasonably performant, and do not have any major ToS concerns (with Vertex AI, data we send in is not used for training and we are allowed to do as we please with output). We will also need a simple web interface to display review comments, but I feel the exact choice of web server package does not matter too much here and we can go with whatever the implementers have the most experience with.
FAQ
Why llvm-zorg?
We discussed amongst ourselves whether we wanted a separate repository or if we want to put code in llvm-zorg. I advocated for putting all of the infrastructure pieces in llvm-zorg given it is the home for all other LLVM infrastructure that does not make sense to have in the monorepo, there is already some existing art in the form of the LLVM bazel bot, and I think using a separate repository adds additional unnecessary friction (e.g., permissions questions around the llvm-wg repo) that we can avoid entirely if we just also use llvm-zorg. I do not foresee this codebase becoming big enough to warrant its own repository either.
Some of these subagents seem more useful than others. Why have all of them?
Alive2 validated miscompiles generated by an LLM are of course likely to be more useful than LLM generated style comments or other forms of pure LLM review. I think the key here is that the information is pluggable, which allows us to experiment with different approaches. If the community wants to have some comments posted automatically upstream, but not others, the infrastructure will fully support that.
Why not use an existing LLM-based review system?
We have tried/seen existing LLM review systems that have native Github Integration such as Gemini, CodeRabbit, and CoPilot. They typically provide very poor-quality reviews, sometimes being outright wrong. We believe the key to good reviews based upon systems like Sashiko is in tailoring prompts (e.g., our coding standards, alive2, llubi) and tooling to LLVM specifically. This is only possible by building out some level of custom infrastructure.
Why upstream?
There is also the question of why doing this under the LLVM umbrella when this is still in the early stages rather than a downstream prototype. I think there are two main reasons for this: productionisation and collaboration. A lot of the ideas (e.g., LLMs to generate test cases that cause crashes/miscompiles) have already been prototyped and have proven to provide real value. Other ideas that we have done some experimentation with (e.g., automated coding standard comments) have shown some promise but need more evaluation. These ideas all share a lot of common infrastructure though and it would be nice to reuse it.
There is also sufficient interest from various groups that I think upstream collaboration makes sense.
