I want to announce the beta availability of a public service for reducing and triaging LLVM test cases. If you have a bunch of test cases you’ve been meaning to reduce, please post a pull request against the repo.
The service can currently handle reducing crashes in llvm tools triggered by IR inputs, crashes in clang triggered by C/C++ inputs, and miscompiles which alive2 can confirm. It does not, and probably will never, handle any failure which requires executing binaries or grepping through results in ad-hoc manners. For those, you’re still on your own.
The other major restriction is that test cases reduced must be fairly short running, and otherwise not require a bunch of resources on the machine doing the reduction. I may loosen those restrictions eventually.
I do need to emphasize that this is currently in early beta. A bunch of it is held together with hacky scripts, cron jobs, and manual work on my part. If this gets a bunch of uptake, please be patient. I’m announcing this now mostly because I need a bunch of test cases to help drive work on reducers, and the easiest way to get those is to publish the repo.
On a technical level, there’s really not too much exciting to this. I’m just wrapping existing reducers, and some ad-hoc python scripting to automate test case reduction. The focus of this project is around the workflow aspects of making it easy to reduce, and encourage the creation of a public corpus of reduced test cases waiting to be fixed. Code can be found here for anyone who wants to see ugly scripting.
Philip