Hi everyone,
I have recently been using a fuzzer I developed to test MLIR, and I have been following the community discussions about fuzzer-discovered issues and AI use. I would like to describe my workflow and ask for feedback on reporting issues and becoming more involved in the community.
Fuzzing and issue reporting
My fuzzer uses traditional fuzzing techniques and does not use an LLM during the fuzzing process. After finding a problem, I use an LLM to help reduce the reproducer, analyze possible causes, and write the issue description based on my own judgment.
The code is still being developed and is not yet open source. I plan to release it in the future.
Since June this year, I have been following and learning about MLIR. I have read parts of the official documentation and material on its design, overall architecture, and pass implementations. In particular, I have continued to read pass-related source code. When analyzing and reporting problems, I try to apply what I have learned and use my own judgment to guide the LLM-assisted analysis and issue writing.
Before filing an issue, I run the reproducer multiple times against the latest revision to confirm that the problem still exists. To reduce duplicate reports, I also look for existing issues and related fixes, and try to understand the conditions that trigger the problem and its possible causes.
Sometimes an earlier issue reports a similar problem and has already received a fix. Its original reproducer no longer triggers the problem, but I can still trigger the same assertion through a similar path. In such cases, I file a new issue based on the information available to me and link the earlier report when possible.
For example, in #237734, I found an affine-related assertion similar to the one in the earlier issue #58419. Before filing, I confirmed that the old reproducer no longer triggered the problem, but the new reproducer still triggered the same assertion, so I filed a new report and linked the earlier issue. However, a crash location or assertion message alone does not always make it possible to determine whether different reproducers have the same root cause.
Despite these checks, I may still misunderstand a cause or miss a related report. Please let me know if you notice a duplicate, incorrect analysis, or anything else I have overlooked.
Possible causes and robustness issues
I would appreciate advice on including possible causes in issue reports. My hope is that such analysis can help maintainers understand and locate a problem, and help future reporters identify similar problems and avoid duplicates. When I cannot confidently assess the cause myself, I am unsure whether it is more helpful to provide only verified information, such as the reproducer, command, and stack trace, or also to include a brief, clearly labeled LLM-assisted hypothesis based on the stack trace and relevant source code.
I would also like to understand which findings should be filed as issues and which do not need to be reported. My current view is that problems causing a compiler crash or assertion failure are generally worth reporting. In particular, some assertion failures may indicate that an assumption made during compilation is not satisfied, even if a build without assertions does not report an error. This could result in output MLIR that does not preserve the original semantics, or in other problems.
At the same time, I am unsure how the community would like to handle robustness issues involving clearly invalid input. For example, in #233053, a negative dimension caused a crash during canonicalization of tensor.expand_shape. The issue was later fixed. I would appreciate guidance on whether cases like this, where the input may violate semantic constraints, should be reported.
Contributing fixes
Besides reporting issues, I would like to gradually contribute fixes and submit PRs. I would like to find a mentor who could offer some guidance as I learn MLIR and work on specific fixes, but I am not sure whom to approach or how to get started.
Questions
- When I cannot confidently assess the cause of a problem, would you prefer a report limited to verified information, or would a brief, clearly labeled LLM-assisted hypothesis also be useful?
- Which kinds of fuzzer-discovered crashes and assertion failures should be reported, and which do not need an issue? In particular, should robustness problems involving inputs that may violate semantic constraints, such as the case in #233053, be reported?
I hope my testing and reports can be useful to the MLIR and LLVM communities. Thank you!