- Do all arithmetic FP operations return a canonical NaN (i.e., a fixed bit pattern)? Or can they return different bit patterns on each execution?
Neither is a correct answer. The basic rule of thumb for operations (note this includes math library functions) in IEEE 754 is the flow is as follows:
- If an input is a sNaN, signal the invalid exception and quiet the NaN.
- If an input is a NaN and the output is a NaN, the result is a qNaN with the same payload as one of the inputs (which one is unspecified, and varies between different hardware implementations).
- If no input is a NaN and the output is a NaN (e.g., 0.0/0.0), a qNaN is returned. I’m not seeing any definition of what the payload can be in IEEE 754, but my understanding is that the intent is that an implementation that stores the address of instruction as the payload in this case is meant to be conforming.
In other words, absent a few special-case operations, it’s guaranteed that the output of an operation is a qNaN, and a “custom” NaN payload is guaranteed to propagate through operations in a matter akin to pointer provenance. There is a concept of “canonical NaN”, but I believe this is parlance for cases like x87’s pseudo-NaN values, and for the basic
fp128 types, every NaN is canonical.
- How instructions always return silent NaNs? Or is there a global flag to make them return signaling NaNs?
With a few exceptions (principally the operations that explicitly only manipulate the sign bit, or that allow users to construct custom NaNs like C’s
setpayload), the result of every arithmetic operation is guaranteed to not be an sNaN.
- Do the regular FP ops in LLVM signal with signaling NaNs? Do you need to use some specific intrinsic? Or does it depend on a global flag?
C explicitly allows implementations the freedom to treat all sNaNs as if they were qNaNs, and C2x adds an
FE_SNANS_ALWAYS_SIGNAL flag to detect if they take this freedom or not. I believe it is reasonable to assume that anything not using
strictfp and constrained intrinsics is operating in an sNaN-is-qNaN mode (in other words, the C2x flag should only be set if
-ffp-model=strict or similar is on the command line).
- Do FP operations canonicalize NaNs? For example, can
fsub %x, 0 be replaced with
%x? If NaN canonicalization happens, then the answer is no! Likewise for
fmul %x, 1.0
The non-constrained operations are assumed to be in default rounding mode and ignoring exceptions, and NaN payloads are guaranteed to propagate, so replacing
fsub %x, 0 with
%x should be legal if the floating-point type is not
ppc_fp128 (which have non-canonical NaNs). Actually, if you’re in a denormals-are-zero function, then it also changes the value, since a denormal is effectively noncanonical in that scenario.
fneg special when handling denormals and NaNs? Do they only flip the sign bit and that’s it?
Yes. These operations are documented as flipping only the sign bit. They don’t even signal an exception if the input is sNaN, and they will return an sNaN output if the input is sNaN. They even preserve noncanonical encodings.
- If 5) is true, then
fsub 0, %x if not equivalent to
That is why an
fneg instruction was effectively added.
- Does load canonicalizes NaNs? This is a long-standing question, so I would like to document it. If the answer is no, can we support x87 correctly? Does that matter? What about bitcast?
- Do phi & select canonicalize NaNs? In case they have fast-math flags, e.g. we only adjust the sign bit (for nsz)?
Canonicalization should ideally happen only at the user’s explicit request, although I believe almost every floating-point arithmetic operation implicitly canonicalizes its inputs. (Note that canonicalization’s effects are to turn sNaN into qNaN, denormals into zero [in DAZ mode], and… something for the
x87_fp80 types and
ppc_fp128 that I doubt too many people actually care about).
I’m not sure if DAZ mode means that
fcmp x, y is equivalent to
fcmp (canonicalize x), (canonicalize y) (it appears to be true on x86 from the manual), but if that is the case, then you can handle noncanonical loads on x87 by only loading it as a float if you know all of the uses implicitly canonicalize, and using integer operations for those uses that don’t canonicalize (which include
fabs). The only case I don’t know how to make noncanonical easily is
ret float %x… maybe you could do it if you used
FXRSTOR? (NB: this is not a performance-oriented lowering.)
- Can the compiler assume any NaN bit pattern. E.g., can we replace
fdiv 0.0, 0.0 with
2139095041 (or any NaN pattern)?
Now you’re getting into what LLVM’s NaN semantics should be as opposed to IEEE 754.
Semantically, any operation that generates a NaN (i.e., none of its inputs were a NaN) should be viewed as producing an unspecified qNaN. Pragmatically, it should be the preferred qNaN (which, as @arsenm noted, is different for older MIPS processors). If the inputs are NaNs, then it is safe to replace
fdiv %x, %y with
canonicalize %x or
canonicalize %y (if both are NaN, it’s unspecified which one it returns, so the optimizer could pick either one). I don’t think it is wise to eliminate user-added
canonicalize operations, but I suspect in practice, we do a bad job of preserving the implicit canonicalize steps of existing FP operations.