Live data from Hacker News

Optimizations Enabled by -ffast-Math

kristerw.github.io

101–110 of 117 posts

Re: Optimizations Enabled by -ffast-Math

#101

Why is this compiler optimization beneficial with -ffinite-math-only and -fno-signed-integers? From if (x > y) { do_something(); } else { do_something_else(); } to the form if (x What happens when x or y are NaN?

> NaN is unordered: it is not equal to, greater than, or less than anything, including itself. x == x is false if the value of x is NaN [0]

My read of this is that comparisons involving NaN on either side always evaluate to false.

In the first one if X or Y is NaN then you'll get do_something_else, and in the second one you'll get do_something.

As far as why one order would be more optimal than the other, I'm not sure. Maybe something to do with branch prediction?

[0] https://www.gnu.org/software/libc/manual/html_node/Infinity-...

Re: Optimizations Enabled by -ffast-Math

#102

Why is this compiler optimization beneficial with -ffinite-math-only and -fno-signed-integers? From if (x > y) { do_something(); } else { do_something_else(); } to the form if (x What happens when x or y are NaN?

It is always false. In the first block do_something_else() gets executed when either is NaN, in the second it is do_something().

Re: Optimizations Enabled by -ffast-Math

#103

Earlier quoted context omitted.

> Opinions obviously vary, but I have never seen any good use case for flush-to-zero. The classical use case is real-time audio, where an IIR filter may have quite slow decay, such that the signal stays in the subnormal regime for many samples. If this happens and your hardware has significant stalls for subnormal data, you may miss your real-time deadline resulting in clicking or other audio corruption.

I agree that this is a good example and there are probably similar cases in video processing and in graphics rendering. However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed. Outside such special cases like real-time DSP and graphics, using flush-to-zero or options like -Ofast or -ffast-math are seldom justifi…

> Outside such special cases like real-time DSP and graphics

Personally I see audio and graphics as the common case, and scientific computing as the special case ;) Do game physics engines expect denormals to be preserved?

Re: Optimizations Enabled by -ffast-Math

#104

Earlier quoted context omitted.

The whole rationale for the standard for floating-point operations is to specify the FP operations with such properties that a naive programmer will be able to write programs which will behave as expected. If you choose any option that is not compliant with the standard, that means, exactly as you have noticed, that you claim that you are an expert in FP computations and you know how to write FP programs that will gi…

Is it adviced to detect overflow and do something about it after an FP computation rather than adding pre-checks to try to avert the overflow?

The only obvious way to test if some operation is going to produce some value(s) is to perform that operation. Yes, for, say, a single multiplication I can anticipate the necessary bounds for each input value. But even short sequences of operations would quickly devolve into labyrinthine nested switch statements.

Re: Optimizations Enabled by -ffast-Math

#105
post #11

I found the following note for -ffinite-math-only and -fno-signed-zeros quite worrying: The program may behave in strange ways (such as not evaluating either the true or false part of an if-statement) if calculations produce Inf, NaN, or -0.0 when these flags are used. I always thought that -ffast-math was telling the compiler. "I do not care about floating point standards compliance, and I do not rely on it. So opti…

> such as not evaluating either the true or false part of an if-statement Do you mean `if-else if` or `if-else`? Because the second would work and would always short-circuit to else.

I imagine it means that the compiler could turn an if(x=6). Which will logically work under these assumptions.

Re: Optimizations Enabled by -ffast-Math

#106

I am not a C dev but this was a really fascinating blog post and how compilers optimize things.

Excellent post I agree, but isn’t it more about how they can’t optimize? Compilers can seem magic how they optimize integer math but this is a great explanation why not to rely on the compiler if you want fast floating point code.

Re: Optimizations Enabled by -ffast-Math

#108

* x+0.0 cannot be optimized to x because that is not true when x is -0.0 Wait, what? What is x + -0.0 then? What are the special cases? The only case I can think of would be 0.0 + -0.0.

Seems like it might depend on the FPU rounding mode?

https://en.wikipedia.org/wiki/Signed_zero#Arithmetic

Re: Optimizations Enabled by -ffast-Math

#109
post #90

Earlier quoted context omitted.

#include #include double interp(double x, double x1, double y1, double x2, double y2) { double node_poly = (x - x1) * (x - x2); double w1 = 1.0 / (x1 - x2); double w2 = 1.0 / (x2 - x1); double y = node_poly * ((w1 * y1) / (x - x1) + (w2 * y2) / (x - x2)); return y; } int main() { for (int p = -300; p >= -330; p -= 1) { double x = pow(10, p); double y = interp(x, 0, 10, 1, 20); printf("x = %.6g, y = %.6g\n", x, y); }…

You implemented the wrong formula. You don’t want to explicitly calculate "node poly". The formula you want is the “second barycentric formula”, (4.2) at the top of page 505 of https://people.maths.ox.ac.uk/trefethen/barycentric.pdf Or formula (2.6) halfway down page 2 of https://people.maths.ox.ac.uk/trefethen/publication/PDF/2011... Also cf. Higham (2004) “The numerical stability of barycentric Lagrange interpolati…

There are advantages to using either version of the formula. If you use the "first" then you do need to take care about overflow/underflow when the number of interpolation points is large, but that isn't an issue here. The "first" behaves better than the "second" for extrapolation, as discussed in the paper by Trefethen et al. And yes, the choice of interpolation points is important.

But none of that should make a difference here. What's actually causing the dubious results jlokier reports isn't exactly proximity to the interpolation points -- I don't think it could happen if one of them weren't exactly zero. What's happening is that the very smallest values of x in jlokier's code are denormal numbers; that is, ones whose floating-point representation has less precision than usual.

Usually a floating-point number looks like 1. x 2^n; for IEEE double precision, the is 52 bits long. So if you start with 1, 1/2, 1/4, 1/8, and keep dividing by 2, you always have stuff=0 and n decreases by 1 each time. But what should you do once you reach the smallest value of n that will fit in the floating-point format? One answer is just to say that after that you get zero. But you can get a little extra dynamic range if instead you say that when n is as small as it's allowed to be, you can use 0. instead of 1.. These numbers have reduced precision for the sake of extra dynamic range, and they are called "denormal". Lots of things go subtly (or not so subtly) wrong when dealing with denormal numbers.

One thing that goes wrong is the interpolation formula we're discussing here. The formula looks like f(x) ~= l(x) . sum of wj f(xj) / (x-xj) where xj are the interpolation points (in this case 0 and 1), l(x) is the product of (x-xj), so in this case x(x-1), and wj = product (xj-xi) where xi ranges over all the other interpolation points besides xj, so w0 = -1 and w1 = +1. So we have f(x) ~= x(x-1) . [-10/x + 20/(x-1)]. In fact this is exact provided x is neither 0 or 1, because the thing we're approximating is a polynomial of degree So, what I claimed before was: suppose x = xj + h where h is small, and suppose you compute l(x) as the product of its factors. Then we get l(x) = h times product (xj+h-xi), and the sum is wj f(xj) / h + the sum of the other terms; and if h is tiny then the other terms in the sum are tiny in comparison, and the factors of h and 1/h cancel and everything's fine.

But that doesn't happen when xj=0 and x (and hence h) is denormal -- because you can't pull the denormal trick at the large end of the range, so the reciprocal of a denormal floating-point number is floating-point infinity!

This isn't a cancellation problem. It isn't a matter of the formula going unstable when x is very close to one of the interpolation points. But it is a problem, and it never occurred to me until jlokier demonstrated it, because denormals are kinda obscure and I didn't think to think about what might happen if they get involved. It would be avoided if you made your interpolator check for values very close to one of the xj. But -- I think, but this stuff is treacherous -- it would also be avoided if you checked for infinities as well as NaNs on the output side, which would be more efficient (provided your inputs are usually sensible). Or -- I think, but this stuff is treacherous -- you could just check explicitly for denormal values of x and maybe of the xj, because if none of those is denormal this pathology can't arise.

Re: Optimizations Enabled by -ffast-Math

#110
post #90

Earlier quoted context omitted.

#include #include double interp(double x, double x1, double y1, double x2, double y2) { double node_poly = (x - x1) * (x - x2); double w1 = 1.0 / (x1 - x2); double w2 = 1.0 / (x2 - x1); double y = node_poly * ((w1 * y1) / (x - x1) + (w2 * y2) / (x - x2)); return y; } int main() { for (int p = -300; p >= -330; p -= 1) { double x = pow(10, p); double y = interp(x, 0, 10, 1, 20); printf("x = %.6g, y = %.6g\n", x, y); }…

You implemented the wrong formula. You don’t want to explicitly calculate "node poly". The formula you want is the “second barycentric formula”, (4.2) at the top of page 505 of https://people.maths.ox.ac.uk/trefethen/barycentric.pdf Or formula (2.6) halfway down page 2 of https://people.maths.ox.ac.uk/trefethen/publication/PDF/2011... Also cf. Higham (2004) “The numerical stability of barycentric Lagrange interpolati…

Thanks for the pointer, and to the paper showing how to calculate error bounds. Interesting paper.
Post reply on HN