for the AFL++ fuzzer project i wrote an llvm pass for transforming floating point
comparisons to equivalent integer comparisons which can be splitted into chains
of byte comparisons. This is a sort of deoptimizing for the purpose of getting
feedback during fuzzing.
While extending the pass for special values like NaN, I came about a strange
behaviour, that might be a bug in the compiler. I am using clang/llvm version 12.0.0.
My test case was like this:
a = NAN;
b = NAN;
assert(!(a <= b));
I expected my pass to encounter a FCMP::OLE instruction during instrumentation.
But with optimization -O3 i got a FCMP::OGT as if i had written
assert((a > b));
Sadly this optimization is not correct for NaN values, which don't compare to anything.
Without optimization (-O0) everything worked as expected with my pass.
Without my pass everything workedalso (with any optimization level), but I don't know why.
So my question is, what needs to be done to use optimization with my pass
and have the right (original) predicate delivered?
Thanks very much,