Notes from the workshop yesterday:
-
Craniumlift - Why Cranelift?
-
ISLE made everything really nice given rules are self contained and easy to verify.
-
Not possible with LLVM currently given Alive2 takes before/after IR rather than a rewrite rule.
-
LPO paper → creating LLVM IR sequences specifically because they can be verified with Alive2
-
What would you keep from the current approach (CraniumLift)
-
LLM generation + verification
-
Looking at extending beyond peephole rewrites
-
-
InstCombine rules are (kind of) written in a DSL
- Having a TableGen based backend would help.
-
Looking at actual production applications
-
Original Alive had a DSL for pattern rewrites? Would help with applying this approach to LLVM.
-
How do we actually get to a declarative peephole optimizer? AI + Formal Verification.
- Before LLMs, pulling out all the peephole rewrites into a DSL would have been a large amount of effort.
-
Do LLM generated patterns run into more problems with ordering?
-
Different people use LLVM in different ways (e.g., one user needs a complex pattern match to fire). How do we get buy-in on the default pipeline?
-
Loading patterns as data files or replacing heuristics with something loadable (e.g., MLGO) can help solve this problem a bit.
-
-
LLM generated peephole rules were pretty bad in some cases.
- How this interacts with combine phase ordering is unclear given Cranelift uses e-graphs.
-
-
AI Policy in LLVM
-
Opinions change
-
Original policy was designed to have some reasonable guardrails but otherwise be very enabling.
-
Proposal recently from nikic@ was to ban LLMs for interpersonal communication.
-
Many people are on board with using LLMs to write code given there is good evidence of benefit.
-
Interpersonal channel communications (code out of scope) should not be created with LLMs.
-
Reid’s opinion is that this is a reasonable thing, enabling LLM use still for documentation (sort of in a gray area around the code/interpersonal communication boundary) is probably good.
-
Clear rules for these different communication channels would probably be helpful.
-
-
What about enforcement?
-
Sometimes it’s really obvious when something is LLM generated.
-
Example of a contributor using LLMs to edit their posts to good effect.
-
What is the cost benefit here? Some people use it well, but we also get a lot of slop.
-
Reid’s opinion is that the cost benefit is positive. It does inconvenience people, but it is worth it for the clarity that the community gets out of it.
-
-
-
Benefits of writing to actually help understanding. Writing a PR description forces you to think about things you otherwise wouldn’t.
-
ESL speakers can benefit greatly from using LLMs in communication.
-
It is important that you can tell whether or not the person you are talking to understands things, especially if you are reviewing code.
-
A lack of clear policy in this space can create situations where members can effectively DOS other contributors by generating slop to counter anything someone trying to hold to a recommendation says.
-
(Comments on one of Chandler’s replies to Nikita’s RFC) Writing/interpersonal communication is actually about community building, which is very much a human thing. There are many insincere ways of communicating, LLM generated text being one of them.
- Writing communication by hand is a signal that someone is actually interested in the community.
-
Anecdote about using LLMs for translation - They work well for directly translating from another language and produce text that sounds human.
-
Anecdote about contributing to a different OSS project, LLM assisted patch, PR description is sort of a cover letter to the project on why they should care about it.
-
Should we be trying to ensure that prompts are available to reviewers?
-
Reviewing a 20k line diff PR is easy. Rejected!
-
If you only have one prompt, your PR is probably not ready.
-
-
LLMs make it very easy to generate a giant amount of slop communication and slop PRs. How do we protect reviewer/maintainer time?
-
If we’re introducing policy specifically to help with this, how do we make it durable against model changes?
-
Two different goals: building community and protecting reviewer/maintainer time. Might require different solutions.
-
-
Maybe we should just treat all of this low quality communication/PRs as a spam problem and use traditional spam moderation tools?
-
If a new model ends up being so good that we cannot discern human vs agent generated communication
-
Is that even a problem?
-
Either we have too many PRs to read and we won’t read them all or we will have too many PRs to read and we will moderate some of them.
-
-
LLVM is built on community. Possible history of rejecting change that might be good for the sake of the community.
-
Vouch system to fight off spam?
-
There are a lot of people who want to be involved in LLVM.
-
E.g., people who fight for good first issues.
-
We need some way to grow people from just wanting to be involved to being able to hold their own/be helpful community members.
-
What do you give to be part of the community? There is lots of pretend give with a lot of take these days with LLM generated PRs.
- Maybe if you just pretend enough you’re actually doing it?
-
-
A lot of these problems are probably made more difficult by the asynchronous nature of communication in LLVM.
- Lots of ways to get synchronous communication. Various scheduled video calls, office hours, local meetups.
-
Healthy communities have some filter, probably with a non-zero false positive rate for rejections, assuming there are alternative ways for motivated people.
-
A decent amount of the culture of LLVM is also driven by the desires of the companies that employ a large number of contributors.
- Having multiple companies here helps a lot here.
-
Costly signals - how do you demonstrate commitment - signal that you are worth investing time into.
- Writing PRs after LLMs are no longer a good signal of commitment here.
-
LLVM is a “manager-free” zone?
-
Idea is that people from different companies can reject PRs without being beholden to whatever corporate pressures are individual to a company.
-
Maybe a reason LLVM has made it as far as it has is a willingness to accept a limited amount of corporate driven tech debt?
-
-
Having someone to help mentor your contributions as a newcomer is extremely useful.
-
Hot take: A vouch system would make LLVM more welcoming. We either spend our time reviewing slop or specifically mentoring people through some formal channel (in an ideal world).
-
The office hours concept is good.
-
A formal pipeline for newcomers to get mentorship might be a good model.
- Having to synchronously join a call is a costly signal (which is a good thing in this instance).
-
-
ML model back to C++ (human understandable heuristic)
-
Black box heuristics are kind of nice in some ways.
-
Some approaches to turn things into decision trees.
-
We can also look at LLMs to generate heuristics, can make heuristics that are more human understandable (although not necessarily similar to a human written heuristic).
- LLMs can touch basically whatever they want. Instead of plugging in a side effect free model, you now have to review in an expensive way all of the code.
-
The bitter lesson - approaches to optimization that get better with more data often win.
-
Can we get some insights out of an ML model?
-
Ablation experiments + approaches that perturb inputs (Lime, shapley values) can show what is important.
-
Data might not be that useful though.
-
-
What about worst case analysis? It can be easier to show that a decision tree will not cause really bad worst case problems than a neural network.
-
-
SWE role shifting from actually writing code to communication with AI.
-
In a couple years will there even be a human role in the development cycle?
-
It seems a safe bet that agents will continue to get better over time.
- You can already chain agents together to get even more gains.
-
Without proper understanding of the codebase and issues, someone is not very useful to the community.
-
It can sometimes be used as a tool to exercise critical thinking and can sometimes provide perspective that pushes you out of potentially incorrect lines of thinking.
-
It’s a rubber duck that talks back to you!
-
LLMs generate plausible looking fixes if you are only somewhat familiar with the codebase. With an LLM generated fix you no longer need to go digging as deep into the codebase which takes away from understanding.
-
-
Roadmap for MLGO
-
Performance modelling is the big problem
-
We have looked at LoopUnroll
-
This is heavily uarch dependent. How do we handle that?
-
Assumptions can be baked into the model, unique model per uarch.
-
Can also feed uarch (parameters) into the model and train on multiple uarches.
-
-
-