There’s been a lot of discussion about code review of LLVM changes recently, in the two recent RFCs about PRs and review AI-generated changes. We at Google have also had a lot of discussions about code review, largely around the security and correctness concerns stemming from possibly unreviewed code.
To get a better understanding of these issues, we’ve created a public dashboard to visualize high-level contribution metrics for the llvm-project repository.
The purpose of this dashboard is to provide the community with data that can help us understand our development processes better and spark constructive conversations. It tracks trends over time, such as:
The daily volume of commits.
A breakdown of commits made via Pull Requests versus those pushed directly to main.
The proportion of PR-based commits that receive reviews before being merged.
Trends in the number of unique contributors.
This is a tool for process improvement, not for judgment. It’s about observing aggregate trends in our shared processes to identify areas where we can improve together.
We believe this data can serve as a helpful, neutral starting point for discussions on topics important to the health of the project, such as:
How can we best support each other to ensure timely reviews?
What are the right workflows for different kinds of changes?
How can we make the contribution process even smoother for new community members?
The dashboard is sourced entirely from publicly available data from the GitHub repository and API.
We welcome your feedback and we look forward to the discussions this may foster as we all work together to ensure the long-term health and success of LLVM.
I wonder if there would be value in splitting the data by subprojects - as someone working only on Clang, I am surprised the unreviewed commit percentage is that high.
What does unreviewed mean exactly? PRs without any comments from people other than llvm-bot and the user who opened it? Unapproved PRs? Commits without a PR?
Glad you like it! We have plans for subsequent releases and improvements on this dash, and subproject breakdown is high on the list. I’ll be sure to post back here when we’ve got new features to announce.
I think tracking this is a good idea.
Here is something which should help to better understand review/contribution:
how long on average does it take a new contributor patch to be reviewed
what about 2nd patch-10th patch from that same contributor?
For new contributor how many revisions of the patch do they go through
For contributions that go through N revisions before approval; what is the time frame between posting the revision and review
How old is the old merge (pull) request that is not approved?
How old is the oldest merge/pull request that does not have any review comments on it?
How many pull requests that are older than a week without a request for changes?
This is definitely not something which is easy to automate but having this will help to better understand if the review process is working overall and can help better understand where improvements are needed.
This is a great start, thank you for working on this.
I would love to see a lot more data. I am thinking about this from my hats that I wear. I have a perspective as someone who spends a lot time looking into regressions as well as someone who spends a lot of time looking at static analysis reports I get internally and thinking about the PRs that generated the code in question.
Some things that would really help us to know are bugs/regressions relative to:
Number of reviewers
Number of comments
Number of comments from different reviewers (Is only one reviewer making comments or are most of them making comments)
Number of days the PR was open
Size of change
Changes made after review (maybe hard to measure?)
Tests added after initial review (maybe hard to measure?)
Test quality (hard to measure?)
This would probably require a step up in discipline of writing bug reports for every bug, which we don’t really do for all cases.
We eventually need a way to track bugs/regressions to changes better and understand what we can do to reduce bugs per PR. Especially if we are talking about understand LLM based code quality, the evidence so far says they lead to a downward pressure on code quality and increase bugs
Commit rate has been pretty stable around 100/day for the past two years. This isn’t surprising, since LLVM is a mature project and has a vibrant community.
Peaks just before releases, and lows during year’s end, as expected.
Unreviewed commits have been steadily reducing, and seems to be half of what it was 2 years ago (40% → 20%).
Bulk of unreviewed commits is pushes directly to main, which would be mostly gone with this change.
It’s unclear if requiring PRs will reduce the number of unreviewed commits, or will just move the bulk to unreviewed PRs merged.
First time contributors per day was slightly larger initially, but hasn’t changed much.
I’m guessing this also includes people who contributed to LLVM before Github, but it’s hard to say how much.
Unique contributors is 3x more than 2 years ago, to 3000, which is a lot!. Not sure how this compares to other upstream projects.
The data shows the growth is steady and healthy. There does not seem to be big problems in growing the community, and the organic nature probably means we’re doing a lot of things right.
To me personally, this seems to confirm we don’t need to address urgent community building issues and, as others have pointed out, the benefits of AI in gaining new contributors would be lower value than the problems it brings. If we had evidence of a dwindling state, we could consider pushing it prematurely, but I don’t believe we need it.
Thanks for putting this together, this is great! One small comment: it’s not super clear how “unreviewed” commits are counted. Are “unreviewed commits” simply commits that don’t go through a pull request? If a pull request is created and then merged before getting a review, does that count as an “unreviewed commit”? I think clarifying the tooltip for Unreviewed Commit Rate might be worthwhile.
I should note open hub is still tracking open source projects (ignore the middle section which is trying to sell you black duck and licensing tools) and have a decent graphs at the end of the page.