Seeking advice on reporting MLIR fuzzer findings and contributing fixes

Hi everyone,

I have recently been using a fuzzer I developed to test MLIR, and I have been following the community discussions about fuzzer-discovered issues and AI use. I would like to describe my workflow and ask for feedback on reporting issues and becoming more involved in the community.

Fuzzing and issue reporting

My fuzzer uses traditional fuzzing techniques and does not use an LLM during the fuzzing process. After finding a problem, I use an LLM to help reduce the reproducer, analyze possible causes, and write the issue description based on my own judgment.

The code is still being developed and is not yet open source. I plan to release it in the future.

Since June this year, I have been following and learning about MLIR. I have read parts of the official documentation and material on its design, overall architecture, and pass implementations. In particular, I have continued to read pass-related source code. When analyzing and reporting problems, I try to apply what I have learned and use my own judgment to guide the LLM-assisted analysis and issue writing.

Before filing an issue, I run the reproducer multiple times against the latest revision to confirm that the problem still exists. To reduce duplicate reports, I also look for existing issues and related fixes, and try to understand the conditions that trigger the problem and its possible causes.

Sometimes an earlier issue reports a similar problem and has already received a fix. Its original reproducer no longer triggers the problem, but I can still trigger the same assertion through a similar path. In such cases, I file a new issue based on the information available to me and link the earlier report when possible.

For example, in #237734, I found an affine-related assertion similar to the one in the earlier issue #58419. Before filing, I confirmed that the old reproducer no longer triggered the problem, but the new reproducer still triggered the same assertion, so I filed a new report and linked the earlier issue. However, a crash location or assertion message alone does not always make it possible to determine whether different reproducers have the same root cause.

Despite these checks, I may still misunderstand a cause or miss a related report. Please let me know if you notice a duplicate, incorrect analysis, or anything else I have overlooked.

Possible causes and robustness issues

I would appreciate advice on including possible causes in issue reports. My hope is that such analysis can help maintainers understand and locate a problem, and help future reporters identify similar problems and avoid duplicates. When I cannot confidently assess the cause myself, I am unsure whether it is more helpful to provide only verified information, such as the reproducer, command, and stack trace, or also to include a brief, clearly labeled LLM-assisted hypothesis based on the stack trace and relevant source code.

I would also like to understand which findings should be filed as issues and which do not need to be reported. My current view is that problems causing a compiler crash or assertion failure are generally worth reporting. In particular, some assertion failures may indicate that an assumption made during compilation is not satisfied, even if a build without assertions does not report an error. This could result in output MLIR that does not preserve the original semantics, or in other problems.

At the same time, I am unsure how the community would like to handle robustness issues involving clearly invalid input. For example, in #233053, a negative dimension caused a crash during canonicalization of tensor.expand_shape. The issue was later fixed. I would appreciate guidance on whether cases like this, where the input may violate semantic constraints, should be reported.

Contributing fixes

Besides reporting issues, I would like to gradually contribute fixes and submit PRs. I would like to find a mentor who could offer some guidance as I learn MLIR and work on specific fixes, but I am not sure whom to approach or how to get started.

Questions

  1. When I cannot confidently assess the cause of a problem, would you prefer a report limited to verified information, or would a brief, clearly labeled LLM-assisted hypothesis also be useful?
  2. Which kinds of fuzzer-discovered crashes and assertion failures should be reported, and which do not need an issue? In particular, should robustness problems involving inputs that may violate semantic constraints, such as the case in #233053, be reported?

I hope my testing and reports can be useful to the MLIR and LLVM communities. Thank you!

Welcome @Kiritobl :wave:,

Thank you for your contributions and for asking! I’ve been seeing an increasing number of bugs reported through fuzzing and find this discussion very timely. First, please also see this discussion:

It would be good if you could contribute there or reuse the proposed infra. IMO, consistency really helps.


Are fuzzer generated issues useful?

In general, my observation is that fuzzer-generated examples tend to capture cases that are unlikely in practice, so I tend to consider those cases lower priority. And since our bandwidth is limited, prioritisation is very important.

That said, those bugs often highlight corner cases that we have been ignoring, as well as blind spots in our designs. So they do bring value.

However, those blind spots often become clearer once a number of similar cases are uncovered. That’s when we notice a pattern and an opportunity for a broader design correction.

To me, an individual ticket is often less useful than understanding that broader pattern.


What format is useful?

The following two examples are really good:

To me, two elements are key:

  • A self-contained reproducer: ideally minimised input, the exact command and the LLVM revision.
  • A clear structure: split the report into sections.

It would be great if everyone used the same format. I suggest introducing a GitHub issue template.


Is LLM generated analysis helpful?

No, not in my view, if it is just unverified output.

I can use an LLM locally. Simply posting its output on GitHub adds noise, IMHO. However, your critical analysis of that output would be very valuable. You already describe applying your own judgment; making clear what you have verified and what remains a hypothesis would help.

Fundamentally, the stack dump already shows where the crash occurs. What I would like to see next is someone engaging with why it happens, how serious the issue is and whether it needs a simple fix or a broader design discussion. For example, the discussion here goes beyond the immediate crash:

What is missing and what can be improved?

IMO, we need to get better at:

  • Tracking and categorising fuzzer-generated issues.

For tracking, we need to make sure that every fuzzer-generated issue is correctly labelled. That’s not always the case. Next, we should make better use of GitHub to get an overview of these issues. This could be a GitHub meta-issue with fuzzer-generated issues as sub-issues. Or a GitHub project. Or both.

For categorisation, we should identify “buckets” of common fuzzer issues and use those to help with triage. Right now, I see these main areas, which can overlap:

  • Valid, but unlikely, input causing a crash.
  • Invalid input that the verifier hasn’t caught, or is unable to catch.
  • Transformation robustness.

I would also suggest assigning issues more consistently to people who intend to work on them, to avoid duplicate work and competing PRs for the same issue. In particular, @Kiritobl, if you report an issue, please make it clear whether you intend to fix it or are reporting it for somebody else to take on.

I hope that this is helpful.

-Andrzej

Thank you, Andrzej, for your helpful feedback and suggestions! I really appreciate you taking the time to share your thoughts. I’ll take your advice into account going forward.

I am of a different opinion on this: of course I can run it locally, but the agent would likely need to run an investigation to explain, it’s not just a question
/answer it may involve the building in debug mode, attaching a debugger or inserting debug probes.
The explanation of how the bug is triggered by walking through the chain of events inside the compiler with code pointers isn’t “noise”: I find it extremely valuable to triage an issue.

I agree that a walkthrough of how the bug is triggered is valuable, including when an LLM helped produce it. Just to clarify what I meant:

I don’t mind folks posting LLM-generated triage results. However, I do expect the contributor to have read, reviewed and understood them, as per our LLVM AI Tool Use Policy. It would help to make clear what they have checked and what remains a hypothesis.

In general, I want to avoid situations where an issue is opened with an LLM-proposed fix and somebody sends a PR implementing it without checking the diagnosis or the reasoning behind that fix. In those cases, the human in the loop step would effectively be missing.

For example, this alone doesn’t tell me whether the contributor has assessed the suggestion:

To avoid that, I prefer people to refrain from simply pasting whatever their favourite LLM tells them, i.e.

<My-favourite-LLM> told me that XYZ should do ABC instead of DEF.

Something like this would be much more helpful:

<My-favourite-LLM> helped me triage this and suggested that XYZ is broken. I checked ABC and DEF, which support that explanation because […]. I’m still unsure about […].`

That gives reviewers something concrete to work with: what the contributor verified, their reasoning, and any remaining uncertainty.

Some of this may sound like semantics, but the human in the loop element is the key part here.

Thank you,
-Andrzej

Thank you, I’ll make it clearer which parts are my own analysis and which are LLM-assisted.

I’ve just uploaded a PR with the proposed template:

Thanks again for starting the discussion!
-Andrzej

I have no skin in this game, but I wonder about one thing. Would it be too unreasonable to ask for a bisect in fuzzer generated reports?

It would have to be “best-effort” since bisect is not always straightforward, but maybe some known-to-fail, known-to-work information would help? As in, bisect to a commit, or explain why you couldn’t.

One would hope that if you have the computing power to fuzz, you would have computing power to bisect too. This would shift the some of the marginal cost of triage from maintainers to the to those running the fuzzers. And it would have no effect on ordinary reports.

Although, can be a double edged sword that annoy the committers. I.e a commit merely exposes the bug and doesn’t cause it. Some food for thought.