Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

51–60 of 233 posts

Re: Beware of Fast-Math

#51

Post by bsenftner »

glkindlmann wrote: post #47
bsenftner wrote: post #44

[flagged]

Forgive them. That kind of educational failure is common: floating point is just not covered in basic intro classes, or covered hurriedly. There are enough subtleties to how floating point code works that even when an elective class covers it, it is still peripheral to the focus of the class (e.g. intro computer graphics). So students can get by without really coming to terms with the ubiquity of rounding error, or t…

This is not an issue of floating point math, it's a basic understanding that nothing comes without a cost. If there is a "faster anything" switch, that is at the cost of something else, and it is our entire professional integrity to understand that relationship, and then investigate what is being compromised by that "optimization". Anything less, that's not engineering.

Re: Beware of Fast-Math

#52

Post by on_the_train »

cycomanic wrote: post #37

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boo…

I worked in cad, robotics and now semiconductor optics. In every single field, floating precision down to the very last digits was a huge issue

Re: Beware of Fast-Math

#53

Post by hansvm »

londons_explore wrote: post #43
cycomanic wrote: post #37

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boo…

It would be nice if there was some syntax for "math order matters, this is the order I want it done in". Then all other math will be fast-math, except where annotated.

The article mentioned that gcc and clang have such extensions. Having it in the language is nice though, and that's the approach Zig took.

Re: Beware of Fast-Math

#54

Post by ajross »

emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

> I get the feeling that the real problem here are the IEEE specs themselves.

Well, all standards are bad when you really get into them, sure.

But no, the problem here is that floating point code is often sensitive to precision errors. Relying on rigorous adherence to a specification doesn't fix precision errors, but it does guarantee that software behavior in the face of them is deterministic. Which 90%+ of the time is enough to let you ignore the problem as a "tuning" thing.

But no, precision errors are bugs. And the proper treatment for bugs is to fix the bugs and not ignore them via tricks with determinism. But that's hard, as it often involves design decisions and complicated math (consider gimbal lock: "fixing" that requires understanding quaternions or some other orthogonal orientation space, and that's hard!).

So we just deal with it. But IMHO --ffast-math is more good than bad, and projects should absolutely enable it, because the "problems" it discovers are bugs you want to fix anyway.

Re: Beware of Fast-Math

#55

Post by storus »

This problem is happening even on Apple MPS with PyTorch in deep learning, where fast math is used by default in many operations, leading to a garbage output. I hit it recently while training an autoregressive image generation model. Here is a discussion by folks that hit it as well:

https://github.com/pytorch/pytorch/issues/84936

Re: Beware of Fast-Math

#56

Post by anthk »

stassats wrote: post #32
eqvinox wrote: post #17

Earlier quoted context omitted.

Those rational numbers fly out the window as soon as your math involves any kind of more complicated trigonometry, or even a square root…

You can turn them back into rationals, (rational (sqrt 2d0)) => 6369051672525773/4503599627370496 Or write your own operations that compute to the precision you want.

My post already covered inexact->exact:

     scheme@(guile-user)> (inexact->exact (sqrt 2.0))
$1 = 6369051672525773/4503599627370496

s9 Scheme fails on this as it's an irrational number, but the rest of Schemes such as STKlos, Guile, Mit Scheme, will do it right.

With Forth (and even EForth if the images it's compiled with FP support), you are on your own to check (or rewrite) an fsqrt function with an arbitrary precision.

Also, on trig, your parent commenter should check what CORDIC was.

https://en.wikipedia.org/wiki/CORDIC

Re: Beware of Fast-Math

#57

Post by goalieca »

Affric wrote: post #36
emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

How does IEEE 754 prevent auto-vectorisation?

Floating point arithmetic is neither commutative or associative so you shouldn’t.

Re: Beware of Fast-Math

#58

Post by [deleted] »

emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

[deleted]

Re: Beware of Fast-Math

#59

Post by anthk »

eqvinox wrote: post #17
anthk wrote: post #3

Earlier quoted context omitted.

On Forth, there's the philosophy of the fixed point: https://www.forth.com/starting-forth/5-fixed-point-arithmeti... With 32 and 64 bit numbers, you can just scale decimals up. So, Torvalds was right. On dangerous contexts (uper-precise medical doses, FP has good reasons to exist, and I am not completely sure). Also, both Forth and Lisp internally suggest to use represented rationals before floating point numbers. Ev…

Those rational numbers fly out the window as soon as your math involves any kind of more complicated trigonometry, or even a square root…

Check CORDIC, please.

https://en.wikipedia.org/wiki/CORDIC

Also, on sqrt functions, even a FP-enabled toy EForth under the Subleq VM (just as a toy, again, but it works) provides some sort of fsqrt functions:

    2 f fsqrt f.
    1.414 ok
Under PFE Forth, something 'bigger':

   40 set-precision ok  
   2e0 fsqrt f. 1.4142135623730951454746218587388284504414 ok
EForth's FP precision it's tiny but good enough for very small microcontrollers. But it wasn't so far from the exponents the 80's engineers worked to create properly usable machinery/hardware and even software.

Re: Beware of Fast-Math

#60

Post by JKCalhoun »

> Even compiler developers can't agree.

> This is perhaps the single most frequent cause of fast-math-related StackOverflow questions and GitHub bug reports

The second line above should settle the first.

Post reply on HN