Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

41–50 of 233 posts

Re: Beware of Fast-Math

#41

Post by kzrdude »

Affric wrote: post #36
emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

How does IEEE 754 prevent auto-vectorisation?

If you write a loop `for x in array { sum += x }` Then your program is a specification that you want to add the elements in exactly that order, one by one. Vectorization would change the order.

Re: Beware of Fast-Math

#42

Post by quotemstr »

All I want for Christmas is a programming language that uses dependant typing to make floating point precision part of the type system. Catastrophic cancellation should be a compiler error if you assign the output to a float with better ulps than you get with worst case operands.

Re: Beware of Fast-Math

#43

Post by londons_explore »

cycomanic wrote: post #37

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boo…

It would be nice if there was some syntax for "math order matters, this is the order I want it done in".

Then all other math will be fast-math, except where annotated.

Re: Beware of Fast-Math

#46

Post by vanderZwan »

Sharlin wrote: post #21
evrimoztamur wrote: post #12

Earlier quoted context omitted.

Does that mean that a physics engine written with these operations will always compile to yield the same deterministic outcomes across different platforms (assuming they correctly implement (or able to do so) algebraic operations)?

It's more like the opposite. These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. So the results may vary depending on what transformations the compiler performs – in particular, they may vary between optimized and non-optimized builds, which normally isn't allowed.

> These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't.

I wonder if it is possible to add an additional constraint that guarantees the transformation has equal or fewer numerical rounding errors. E.g. for floating point doubles (0.2 + 0.1) - 0.1 results in 0.20000000000000004, so I would expect that transforming some (A + B) - B to just A would always reduce numerical error. OTOH, it's floating point maths, there's probably some kind of weird gotcha here as well.

Re: Beware of Fast-Math

#47

Post by glkindlmann »

bsenftner wrote: post #44

[flagged]

Forgive them. That kind of educational failure is common: floating point is just not covered in basic intro classes, or covered hurriedly. There are enough subtleties to how floating point code works that even when an elective class covers it, it is still peripheral to the focus of the class (e.g. intro computer graphics). So students can get by without really coming to terms with the ubiquity of rounding error, or the relationship between denormalized and normalized values. Understanding those fundamentals is a pre-req to understanding how corners are being cut with fast-math.

Re: Beware of Fast-Math

#48

Post by stingraycharles »

kzrdude wrote: post #41
Affric wrote: post #36

Earlier quoted context omitted.

How does IEEE 754 prevent auto-vectorisation?

If you write a loop `for x in array { sum += x }` Then your program is a specification that you want to add the elements in exactly that order, one by one. Vectorization would change the order.

Yup, because of the imprecision of floating points, cannot just assume that “(a + c) + (b + d)” is the same as “a + b + c + d”.

It would be pretty ironic if at some point fixed point / bignum implementations end up being faster because of this.

Re: Beware of Fast-Math

#49

Post by glkindlmann »

orlp wrote: post #4

I helped design an API for "algebraic operations" in Rust: https://github.com/rust-lang/rust/issues/136469 >, which are coming along nicely. These operations are 1. Localized, not a function-wide or program-wide flag. 2. Completely safe, -ffast-math includes assumptions such that there are no NaNs, and violating that is undefined behavior . So what do these algebraic operations do? Well, one by itself doesn't do much…

That sounds neat. What would be really neat is if the language helped to expose the consequences of the ensuing rounding error by automating things that are otherwise clumsy for programmers to do manually, like running twice with opposite rounding directions, or running many many times with internally randomized directions (two of the options in Sec 4 of *). That is, it would be cool if Rust enabled people learn about the subtleties of floating point, instead of hiding them away.

* https://people.eecs.berkeley.edu/~wkahan/Mindless.pdf

Re: Beware of Fast-Math

#50

Post by legobmw99 »

vanderZwan wrote: post #46
Sharlin wrote: post #21

Earlier quoted context omitted.

It's more like the opposite. These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. So the results may vary depending on what transformations the compiler performs – in particular, they may vary between optimized and non-optimized builds, which normally isn't allowed.

> These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. I wonder if it is possible to add an additional constraint that guarantees the transformation has equal or fewer numerical rounding errors. E.g. for floating point doubles (0.2 + 0.1) - 0.1 results in 0.20000000000000004, so I would expect that transforming some (A + B…

Kahan summation is an example (also described in the top level article) of one such “gotcha”. It involves adding a term that - if floats were algebraic in this sense - would always be zero, so ffast-math often deletes it, but this actually completely removes the accuracy improvement of the algorithm
Post reply on HN