We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator).
A previous testing effort, using Csmith, found 202 LLVM bugs, which represented 2% of all reported bugs at that time (PDF: https://www.flux.utah.edu/download?uid=114): . However, after this paper was published we are unaware of any further testing using Csmith, and we would like to ask you if you are aware of any such efforts or further results.
Best regards,
Radu Ometita,
Functional compilers engineer @IOHK
Zhendong, who’s done a lot of work on automated testing of llvm.
Hello everyone!
We are working on writing a paper about testing the reliability of C compilers by using Csmith (a random C99 program generator).
A previous testing effort, using Csmith, found 202 LLVM bugs, which represented 2% of all reported bugs at that time (PDF: https://www.flux.utah.edu/download?uid=114): . However, after this paper was published we are unaware of any further testing using Csmith, and we would like to ask you if you are aware of any such efforts or further results.
Just speaking for myself here, I use Csmith as part of my pre-commit testing.
There’s also a couple other active fuzzing efforts running concurrently:
OSSFuzz is continuously running a set of codegen and individual pass fuzzers. A bit more information can be found here:
We (Azul Systems) have a continuously running Java fuzzer which exercises LLVM through our Falcon compiler, and regularly finds regressions and occasional deep long lurking issues. We don’t have a public bug tracker for this, but a sizable portion of our upstream bug fixing activity is driven by the output of this tool.
You may also be interested in the following resources on compilers correctness (articles, software, and talks – from the general topics to the ones specifically focused on testing, validation, and verification):