If the parameter passed is NaN in this example, the result of cmp1 is false and
the result of cmp2 is true. In this case, the result is completely opposite.
When will we get ‘fcmp ugt’ and when will we get ‘fcmp ugt’ ? When comparing
ordered and unorder can be converted ? Such as converting an ordered less than
“setolt” into unordered greater than “setuge” operation.

I assume your question is actually “when will we get fcmp ogt and when will we get fcmp ugt?”

Most languages prescribe that their comparisons are all ordered. So when lowering an operation like a < b, the front-end will generate a fcmp olt. The C language does not prescribe any way to get an unordered comparison, and so the frontend will never directly generate an unordered operation.

Where unordered floating-point comparisons are generated is most frequently by optimizations. Note that there are four possible states for comparing floating point numbers: unordered, equal, less-than, or greater-than. The fcmp instruction has 16 possible comparisons, which will return true for a certain subset of those comparisons (with there being a useless fcmp true and fcmp false variant that unconditionally returns true or false). Negating an fcmp instruction is equivalent to using an fcmp comparison that flips the true/false state: if you negate a fcmp that returns true for only the less-than state, that is equivalent to an fcmp that returns true for unordered, equal, or greater-than. In other words, not (fcmp olt) is equivalent to fcmp uge.

Your mention of setogt/setule makes me think that you are interested in the interaction with SelectionDAG. SelectionDAG actually supports three different floating-point comparison types: ordered, unordered, and don’t-care. The latter class appears when the fcmp instruction has the nnan fast-math flag set, appears as a setgt SDNode instead of setogt or setugt, and can validly be implemented by either an unordered or ordered comparison instruction, whichever is less expensive to implement in hardware. If you’re implementing a backend, carefully check the documentation of your hardware to see what the behavior of instructions is when presented with NaNs.

IEEE 754 standard has a strict definition on comparison predicates. In short, symbol “<=” is not the flip of “>”. There’s no such symbol in C language currently (Maybe in draft, I’m not sure), but LLVM IR does support all predicates. So the opposition of predicates ogt (“>” is C) is ule and the opposition of predicates ole (“<=” is C) is ugt.

That’s why you get opposite results from cmp1 and cmp2. The equivalent to cmp1 in IR is: