A longstanding project on the LLVM "open projects" page is to run random C code through random LLVM passes.
We have found many LLVM bugs by doing the first of these, but not the second: we test only -O[0123s].
Slowly but surely, LLVM is evolving resistance to our random tester. Of course this is good, but it means that we need to keep expanding our efforts if we want to continue to get solid return on investment for our bugfinding hours.
My first question is: When we start running random combinations of passes using opt, which ones should we leave out, because you folks don't want to hear that they're wrong? For example, the other day I learned that ABCD is not yet ready for this sort of testing. A while ago someone on the list said something similar about predsimplify (though it looks like that pass has disappeared from the tree?).
Also I'd appreciate some advice about probabilities for each pass. For example, it seems like a lot of passes will have little to do if Mem2Reg doesn't run, so perhaps that pass should get >0.5 probability of running. Does that make sense and do other examples come to mind?
Frankly I don't expect a zillion bugs to come out of this but it is super easy and you asked for it :).
A longstanding project on the LLVM "open projects" page is to run random
C code through random LLVM passes.
We have found many LLVM bugs by doing the first of these, but not the
second: we test only -O[0123s].
Slowly but surely, LLVM is evolving resistance to our random tester. Of
course this is good, but it means that we need to keep expanding our
efforts if we want to continue to get solid return on investment for our
bugfinding hours.
My first question is: When we start running random combinations of
passes using opt, which ones should we leave out, because you folks
don't want to hear that they're wrong? For example, the other day I
learned that ABCD is not yet ready for this sort of testing. A while
ago someone on the list said something similar about predsimplify
(though it looks like that pass has disappeared from the tree?).
I think that if a pass is not part of -O[0123s] or -std-compile-opts, or
-std-link-opts
then it is not ready for random testing.
Also I'd appreciate some advice about probabilities for each pass. For
example, it seems like a lot of passes will have little to do if Mem2Reg
doesn't run, so perhaps that pass should get >0.5 probability of
running. Does that make sense and do other examples come to mind?
You should have loops and nested loops (in some tests), when you want to
test loop optimizers.
You should have some tests with EH, some without.
Also I think you should also try to test LLVM IR features that are less
often used:
byval, weak linkage, ssp, structs with unusual layout/alignment, vector
operations,
multiple return values, indirectbr, atomic ops, lifetime/invariant
annotations, etc.
My first question is: When we start running random combinations of passes using opt, which ones should we leave out, because you folks don't want to hear that they're wrong? For example, the other day I learned that ABCD is not yet ready for this sort of testing. A while ago someone on the list said something similar about predsimplify (though it looks like that pass has disappeared from the tree?).
check out include/llvm/Support/StandardPasses.h. All of the passes scheduled
there need to be solid because they are really being used.
Also I'd appreciate some advice about probabilities for each pass. For example, it seems like a lot of passes will have little to do if Mem2Reg doesn't run, so perhaps that pass should get >0.5 probability of running. Does that make sense and do other examples come to mind?
You could start with the list of passes you get at -O3, then randomly
alter it, shifting the pass order, dropping passes and introducing more.
Each run might take too long though due to scheduling many passes.
Frankly I don't expect a zillion bugs to come out of this but it is super easy and you asked for it :).
Thanks Duncan and Edwin-- looking at the passes used everyday sounds like the right approach.
Also I think you should also try to test LLVM IR features that are less
often used:
byval, weak linkage, ssp, structs with unusual layout/alignment, vector
operations,
multiple return values, indirectbr, atomic ops, lifetime/invariant
annotations, etc.
Well, we are actually testing C compilers, not specifically LLVM, so we're not going to generate anything that clang or llvm-gcc doesn't use already.
We do generate packed structs, bitfields, etc. in an attempt to stress some of these things.