Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Optimizations Enabled by -ffast-Math

kristerw.github.io

Post reply on HN
Print view View original

81–90 of 117 posts

Re: Optimizations Enabled by -ffast-Math

#81

Post by twic »

jacobolus wrote: post #53
pinteresting wrote: post #43

Earlier quoted context omitted.

Can you think of a function where the input is valid, the output is NaN and nothing has gone wrong in the process? I can't think of any, haven't experienced any, not heard of any examples of it, so you're welcome to break my ignorance on the subject.

This depends how you define “gone wrong”. In evaluating rational functions (a useful tool for approximating all sorts of other functions), one efficient and well behaved algorithm is the “barycentric formula”, based on interpolating function values at various specific input points. When the input is one of those points directly, this formula results in Inf / Inf = NaN. Evaluation code needs to check for this case, an…

What is the rationale for liking rational functions?

Years ago i had some interpolation to do, and used the barycentric rational from Boost [1]. The resulting curve was all over the place. A natural cubic spline gave a much more subjectively sensible-looking fit.

[1] https://www.boost.org/doc/libs/1_77_0/libs/math/doc/html/mat...

Re: Optimizations Enabled by -ffast-Math

#82

Post by adrian_b »

stephencanon wrote: post #76
adrian_b wrote: post #44

Earlier quoted context omitted.

Your arguments are correct, but the conclusion does not result from them. If we assume that underflows happen in your program and this, as you say, is a sign that greater problems will be caused by that, then you must not enable flush-to-zero, but you must enable trap-on-underflow, to see where underflows happen and to investigate the reason and maybe rearrange your formulas to avoid the too small results. Flush-to-z…

> Opinions obviously vary, but I have never seen any good use case for flush-to-zero. The classical use case is real-time audio, where an IIR filter may have quite slow decay, such that the signal stays in the subnormal regime for many samples. If this happens and your hardware has significant stalls for subnormal data, you may miss your real-time deadline resulting in clicking or other audio corruption.

I agree that this is a good example and there are probably similar cases in video processing and in graphics rendering.

However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed.

Outside such special cases like real-time DSP and graphics, using flush-to-zero or options like -Ofast or -ffast-math are seldom justified and they can be dangerous, especially when they are used without a prior analysis and tests to determine whether exception conditions really appear when the program is run, and why.

Re: Optimizations Enabled by -ffast-Math

#83

Post by stephencanon »

adrian_b wrote: post #82
stephencanon wrote: post #76

Earlier quoted context omitted.

> Opinions obviously vary, but I have never seen any good use case for flush-to-zero. The classical use case is real-time audio, where an IIR filter may have quite slow decay, such that the signal stays in the subnormal regime for many samples. If this happens and your hardware has significant stalls for subnormal data, you may miss your real-time deadline resulting in clicking or other audio corruption.

I agree that this is a good example and there are probably similar cases in video processing and in graphics rendering. However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed. Outside such special cases like real-time DSP and graphics, using flush-to-zero or options like -Ofast or -ffast-math are seldom justifi…

> However not all CPUs stall for subnormal data and if the CPU designers would have ensured that none of them stall, no such discussions would have been ever needed.

I don't really disagree (I've pushed hard for no subnormal stalls for years), but there are some timing and area tradeoffs that come with this that make it a somewhat hard sell for FPUs that aren't built entirely around FMA (there are tradeoffs even for designs that are purely FMA based, but they're a lot smaller).

Mainstream x86 designs _still_ have some subnormal stalls!

Re: Optimizations Enabled by -ffast-Math

#84

Post by jlokier »

adrian_b wrote: post #65
jlokier wrote: post #61

Earlier quoted context omitted.

> When the input is one of those points directly, That's against one of the rules of well-behaved floating point programming: Never test for equality.

Like any rule, this has exceptions. You should never test for equality numbers which are known only approximately, which is the case for most FP numbers. There are nonetheless cases when certain FP values are known exactly and it is OK to test them for equality. In general the rule does not depend on number representation, it applies equally to floating-point numbers, fixed-point numbers, rational numbers and even la…

There are exceptions.

But the type of barycentric interpolation formula that leads to Inf/Inf at an interpolation point isn't one of those exceptions - it's an example which shows why the rule should be borne in mind.

When the input is very close to one of the interpolation points, the calculation becomes ill-conditioned, unsurprisingly as two values are diverging towards Inf, so the result can become incorrect as the input approaches the interpolation point, until it becomes Inf/Inf = Nan at some even closer, but non-zero, distance. Before reaching Nan it can also have an interval where the result is +/-Inf.

It depends on the interpolation points. But under some circumstances, it is necessary to recognise when the input is close to an interpolation point, and adjust the formula appropriately.

Re: Optimizations Enabled by -ffast-Math

#85

Post by beagle3 »

pinteresting wrote: post #29
rocqua wrote: post #11

I found the following note for -ffinite-math-only and -fno-signed-zeros quite worrying: The program may behave in strange ways (such as not evaluating either the true or false part of an if-statement) if calculations produce Inf, NaN, or -0.0 when these flags are used. I always thought that -ffast-math was telling the compiler. "I do not care about floating point standards compliance, and I do not rely on it. So opti…

Generally if you're seeing a NaN/Inf something has gone wrong, It's very difficult to gracefully recover from and if you tried I think you would lose both sanity and performance! Regarding performance, the cost of a real division is about 3-4 orders worse performance than an if statement that is very consistent, but the usual way is to have fast/safe versions of functions, where you need performance and can deduce if…

I often use NaNs on purpose to designate “unknown”, and its properties of “infecting” computations mean that I don’t have to check every step - just at the end of the computation.

R goes even farther, and uses one specific NaN (out of the million-billions) to signal “not available”, while other NaNs are just NaNs.

Its properties, used properly, make code much more simple and sane.

Re: Optimizations Enabled by -ffast-Math

#86

Post by PaulDavisThe1st »

ptidhomme wrote: post #5

Do some applications actually use denormal floats ? I'm curious.

If you want maximum possible accuracy for FP computation, you need to pay the performance price that denormals incur. In many cases, that will likely be a long running computation where responsiveness from a user-perspective and/or meeting realtime deadlines is not an issue. Any time "I want the best possible answer that we can compute", you should leave denormals enabled.

By contrast, in media software (audio and/or video) denormals getting flushed to zero is frequently necessary to meet performance goals, and ultimately has no impact on the result. It's quite common for several audio FX (notably reverb) to generate denormals as the audio fades out, but those values are so far below the physical noise floor that it makes no difference to flush them to zero.

Re: Optimizations Enabled by -ffast-Math

#87

Post by jlokier »

gjm11 wrote: post #75
jlokier wrote: post #61

Earlier quoted context omitted.

> When the input is one of those points directly, That's against one of the rules of well-behaved floating point programming: Never test for equality.

It's worth being a bit more explicit about this. The appearance of NaNs doesn't result from breaking the "never test for equality" rule. It happens when the point where you're evaluating the function just happens to exactly match one of the interpolation points. But if you decide to deal with the NaN issue by (1) testing whether your input exactly matches any of the interpolation points, or (2) testing the result for…

> There's no catastrophic cancellation or anything. And floating point has much more dynamic range than precision, so to speak, so until h literally reaches zero you aren't running into trouble.

I just tried it to be sure, in C with "double", and organized the calculation the way it's described in the paper you cited, formula 2.1.

It failed due to catastrophic rational cancellation close to an interpolation point, depending on the values involved. An equality test was not enough.

Approaching the first interpolation point, before it transitioned from the correct result, there were some incorrect Inf results prior to it switching to Nan for an input not equal to any interpolation point.

This was trivial 1d interpolation between two points, where the points were (0.0 => 10.0, 1.0 => 20.0).

Re: Optimizations Enabled by -ffast-Math

#88

Post by kloch »

pinteresting wrote: post #29
rocqua wrote: post #11

I found the following note for -ffinite-math-only and -fno-signed-zeros quite worrying: The program may behave in strange ways (such as not evaluating either the true or false part of an if-statement) if calculations produce Inf, NaN, or -0.0 when these flags are used. I always thought that -ffast-math was telling the compiler. "I do not care about floating point standards compliance, and I do not rely on it. So opti…

Generally if you're seeing a NaN/Inf something has gone wrong, It's very difficult to gracefully recover from and if you tried I think you would lose both sanity and performance! Regarding performance, the cost of a real division is about 3-4 orders worse performance than an if statement that is very consistent, but the usual way is to have fast/safe versions of functions, where you need performance and can deduce if…

NaN usually indicates a bug or algorithmic problem, but not always. I have had cases where I need to detect NaN, present a useful message to the user even though everything is working as designed.

Re: Optimizations Enabled by -ffast-Math

#89

Post by gjm11 »

jlokier wrote: post #87
gjm11 wrote: post #75

Earlier quoted context omitted.

It's worth being a bit more explicit about this. The appearance of NaNs doesn't result from breaking the "never test for equality" rule. It happens when the point where you're evaluating the function just happens to exactly match one of the interpolation points. But if you decide to deal with the NaN issue by (1) testing whether your input exactly matches any of the interpolation points, or (2) testing the result for…

> There's no catastrophic cancellation or anything. And floating point has much more dynamic range than precision, so to speak, so until h literally reaches zero you aren't running into trouble. I just tried it to be sure, in C with "double", and organized the calculation the way it's described in the paper you cited, formula 2.1. It failed due to catastrophic rational cancellation close to an interpolation point, de…

Hm, interesting. I also tried some numerical experiments to sanity-check and I got reasonable results right up to 1ulp away from the interpolation points. I tried again using exactly your example -- f(x) = 10+10x, interpolation points at 0 and 1 -- and again everything seemed fine.

I wonder what we're doing differently, or whether we happened to test at different points. Here's my code:

    def interpolate1(f,x,xs,ws):
        p,s = 1,0
        for (xj,wj) in zip(xs,ws):
            p *= x-xj
            s += wj*f(xj)/(x-xj)
        return p*s
(it's "interpolate1" because I also implemented the other version of the formula described in the article, which for this purpose behaves very similarly).

For your example we want f(x) = 10+10x, xs=[0,1], ws=[-1,1].

As I mentioned before, if you implement the calculation of the polynomial called l(x) in the paper -- the thing my code calls p -- in an unfortunate way, you can get cancellation problems, but for this case I'm not sure I see any way for that to happen.

I tried calculating l(x) in a different way that could conceivably cause cancellation issues, aimed at this particular interpolant:

    def foo(x):
        p = x*x-x
        s = -1*10/x + 1*20/(x-1)
        return p*s
but this also gave reasonable results for every value of x I tried other than exactly 0 and exactly 1.

What exact values of x are giving you what bad results? What language are you implementing this in, with what data types?*

Re: Optimizations Enabled by -ffast-Math

#90

Post by jlokier »

gjm11 wrote: post #89
jlokier wrote: post #87

Earlier quoted context omitted.

> There's no catastrophic cancellation or anything. And floating point has much more dynamic range than precision, so to speak, so until h literally reaches zero you aren't running into trouble. I just tried it to be sure, in C with "double", and organized the calculation the way it's described in the paper you cited, formula 2.1. It failed due to catastrophic rational cancellation close to an interpolation point, de…

Hm, interesting. I also tried some numerical experiments to sanity-check and I got reasonable results right up to 1ulp away from the interpolation points. I tried again using exactly your example -- f(x) = 10+10x, interpolation points at 0 and 1 -- and again everything seemed fine. I wonder what we're doing differently, or whether we happened to test at different points. Here's my code: def interpolate1(f,x,xs,ws): p…

  #include 
  #include 

  double interp(double x, double x1, double y1, double x2, double y2)
  {
      double node_poly = (x - x1) * (x - x2);
      double w1 = 1.0 / (x1 - x2);
      double w2 = 1.0 / (x2 - x1);
      double y = node_poly * ((w1 * y1) / (x - x1) + (w2 * y2) / (x - x2));
      return y;
  }

  int main()
  {
      for (int p = -300; p >= -330; p -= 1) {
          double x = pow(10, p);
          double y = interp(x, 0, 10, 1, 20);
          printf("x = %.6g, y = %.6g\n", x, y);
      }
      return 0;
  }
Post reply on HN