Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Examples of floating point problems

jvns.ca

Post reply on HN
Print view View original

171–179 of 179 posts

Re: Examples of floating point problems

#171

Post by alehlopeh »

maximilianburke wrote: post #112
BeetleB wrote: post #109

Earlier quoted context omitted.

> Both C and C++. Seems C++ only added it in C++11. Surprising. > That's why I said in the second part that we need to do away with them. Do away entirely with fast math calculations? That would be horrible. They exist for a very good reason: Some applications are too slow without them. Enabling subnormal numbers can really slow things down. I'd wager that for the majority of programs written, the fast math is as goo…

> I'd wager that for the majority of programs written, the fast math is as good as the accurate math. I'd take that wager. I spent 13 years working on video games and video game technology, a domain where floating point performance is critical, and by and large we never used fast-math because of the problems it created for us.

I’m more curious about how it caused problems for you if you ever used it

Re: Examples of floating point problems

#172

Post by CamperBob2 »

toolslive wrote: post #159
CamperBob2 wrote: post #144

Earlier quoted context omitted.

What would be an example of a floating-point addition operation that doesn't commute?

>>> x = 7.00000000000001 >>> x + 1 - x 1.0000000000000009 >>> x - x + 1 1.0

I may be overlooking your point. x-x+1 is evaluated left to right like any other additive expression, and after the x-x part is evaluated, the intermediate result is zero. This would be the case with any numeric type, wouldn't it?

Re: Examples of floating point problems

#173

Post by GuB-42 »

toolslive wrote: post #159
CamperBob2 wrote: post #144

Earlier quoted context omitted.

What would be an example of a floating-point addition operation that doesn't commute?

>>> x = 7.00000000000001 >>> x + 1 - x 1.0000000000000009 >>> x - x + 1 1.0

Addition is commutative with floating point numbers, what you show is actually an associativity problem in disguise.

Subtraction is obviously not commutative, so let's rewrite as an addition.

  x + 1 - x => x + 1 + (-x)
  x - x + 1 => x + (-x) + 1
Now write the implicit parenthesis

  x + 1 + (-x) => (x + 1) + (-x)
  x + (-x) + 1 => (x + (-x)) + 1
Now since we assumed addition is commutative, let swap a few things

  (x + 1) + (-x) => (-x) + (x + 1)
  (x + (-x)) + 1 => ((-x) + x) + 1
And you get the classic associativity problem. If, by swapping things, we would have gotten the same expression, it would have been a contradiction, proving that addition was not commutative, but it is not the case here.

It means that if addition is not associative, which we know is the case, then your example doesn't prove that addition is not commutative.

Of course, I didn't prove that in general, addition is commutative, but it has no reason not to be.

Re: Examples of floating point problems

#174

Post by sacrosancty »

defrost wrote: post #132
svat wrote: post #120

Earlier quoted context omitted.

That's exactly my point: when you internalize the diagram, you'll be able to reason confidently about what happens: • In the case of "2.3 + 2.3", each "2.3" is “snapped” to the nearest representable value (green line in the diagram), then their sum snapped to the nearest green line. In this case, because the two summands are equal and we're using binary floating-point, the result will also be a green line. If you kne…

> Again, with the mental model, you'll be in a better position to state what you expect by "exactly". I'm long in the tooth, if I want exactly I'll use a symbolic algebra program (such as Cayley|Magma). I don't expect exactly when using floats (or doubles, etc) and my mental model includes the image of the "fine teeth of coverage" having uneven spacing, that the float graticule across the reals is uneven and at best…

[dead]

Re: Examples of floating point problems

#175

Post by toolslive »

GuB-42 wrote: post #173
toolslive wrote: post #159

Earlier quoted context omitted.

>>> x = 7.00000000000001 >>> x + 1 - x 1.0000000000000009 >>> x - x + 1 1.0

Addition is commutative with floating point numbers, what you show is actually an associativity problem in disguise. Subtraction is obviously not commutative, so let's rewrite as an addition. x + 1 - x => x + 1 + (-x) x - x + 1 => x + (-x) + 1 Now write the implicit parenthesis x + 1 + (-x) => (x + 1) + (-x) x + (-x) + 1 => (x + (-x)) + 1 Now since we assumed addition is commutative, let swap a few things (x + 1) + (…

You're right. it's non-associativity in disguise. You might not like the next example neither:

    NaN + 1  1 + NaN
(as NaN NaN)

IIRC it also breaks down with signed zeros.

Re: Examples of floating point problems

#176

Post by svnpenn »

Here is Go version. works exactly as expected, no surprises. People just need to grow up and use a modern language, not a 50 year old out of date language:

    package main
    
    import "fmt"
    
    func main() {
       var i, iterations, meters float64
       for iterations = 100_000_000; i

Re: Examples of floating point problems

#177

Post by gnufx »

zokier wrote: post #167
gnufx wrote: post #164

Earlier quoted context omitted.

Why is it crazy? Some of us don't want to lose a factor of two on linear algebra (and also care about correctness). I remember testing for correctness against Kahan's tests after FMA became available in RS/6000.

If you do want the precision improvement of fma, then it makes far more sense to explicitly call fma instead of relying compiler on doing transformation that might not happen for any number of reasons. The key here is predictability, if it was actually guaranteed that expressions in the form of (x*y) + z are always done with fma, then it'd be less crazy. But now you have no way of knowing without looking at the produ…

As I implied, we're normally interested in FMA for speed, not numerical properties. I don't know in what circumstances GCC wouldn't use it when vectorizing, but I haven't seen them.

Re: Examples of floating point problems

#178

Post by ilyt »

kstenerud wrote: post #154
ilyt wrote: post #121

Earlier quoted context omitted.

> But let's study it a bit. Suppose you are searching an array for a value, and the value is not in the array. What do you return for an index into the array? People often use -1 as the "not found" value. But then what happens when the -1 value is not noticed? It winds up corrupting further attempts to use it. The problem is that integers do not have a NaN value to use for this. You return (value, found), or (value,e…

> You return (value, found), or (value,error) or Result . And this is great for environments that can support it, but as the levels get lower and lower, such safety nets become prohibitively expensive. Take data formats, for example. Say we have a small device that records ieee754 binary float32 readings. A simple format might be something like this: record = reading* terminator; reading = float(32, ~) | invalid; inv…

> And this is great for environments that can support it, but as the levels get lower and lower, such safety nets become prohibitively expensive.

I don't think there is low enough level here anymore. $0.3 micros maybe ? Why you're doing floats on them ? $2-3 micros already can have 64kB flash. You're just inventing imaginary cases to support "let's just do it worse and uglier" case.

Re: Examples of floating point problems

#179

Post by thxg »

gnufx wrote: post #165
thxg wrote: post #62

Earlier quoted context omitted.

> It is pretty crazy imho that gcc defaults to using fma Yes! Different people can make different performance-vs-correctness trade-offs, but I also think reproducible-by-default would be better. Fortunately, specifying a proper standard (e.g. -std=c99 or -std=c++11) implies -ffp-contract=off. I guess specifying such a standard is probably a good idea independently when we care about reproducibility. Edit: Thinking ab…

GCC doesn't default to non-conforming behaviour like -ffast-math -- that's Intel (at least a similar option). That's usually why people mistakenly think GCC vectorization is deficient if they don't use -funsafe-math-optimizations in particular.

Indeed GCC does not enable -ffast-math by default. Unfortunately, -ffast-math and -funsafe-math-optimizations (despite the name) are not the only options that prevent bit-for-bit-reproducible floating point. For example, -ffp-contract=fast is enabled by default [1], and it will lead to different floating-point roundings: Compare [2] which generates an FMA instruction, to [3] when -std=c99 is specified. As another example, -fexcess-precision=fast is also enabled by default. Similarly, [4] does intermediate calculations in the 80-bit x87 registers, while [5] has additional loads and stores to reduce the precision of intermediate results to 64 bits. In both examples, GCC generates code that does not conform to IEEE-754, unless -std=c99 is specified.

[1] From the man page:

    -ffp-contract=style
           -ffp-contract=off disables floating-point expression
           contraction.  -ffp-contract=fast enables floating-point
           expression contraction such as forming of fused multiply-
           add operations if the target has native support for them.
           -ffp-contract=on enables floating-point expression
           contraction if allowed by the language standard.  This is
           currently not implemented and treated equal to
           -ffp-contract=off.
           
           The default is -ffp-contract=fast.
[2] https://godbolt.org/z/GKb7G4nW9

[3] https://godbolt.org/z/KTnqcT6aW

[4] https://godbolt.org/z/4q31oEe14

[5] https://godbolt.org/z/qdf4hceca

Post reply on HN