Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

31–40 of 233 posts

Re: Beware of Fast-Math

#31

Post by emn13 »

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in, not opt out. And at the very least, standards need to evolve to support hardware realities of today.

Not being able to auto-vectorize seems like a pretty critical bug given hardware trends that have been going on for decades now; on the other hand sacrificing platform-independent determinism isn't a trivial cost to pay either.

I'm not familiar with the details of OpenCL and CUDA on this front - do they have some way to guarrantee a specific order-of-operations such that code always has a predictable result on all platforms and nevertheless parallelizes well on a GPU?

Re: Beware of Fast-Math

#32

Post by stassats »

eqvinox wrote: post #17
anthk wrote: post #3

Earlier quoted context omitted.

On Forth, there's the philosophy of the fixed point: https://www.forth.com/starting-forth/5-fixed-point-arithmeti... With 32 and 64 bit numbers, you can just scale decimals up. So, Torvalds was right. On dangerous contexts (uper-precise medical doses, FP has good reasons to exist, and I am not completely sure). Also, both Forth and Lisp internally suggest to use represented rationals before floating point numbers. Ev…

Those rational numbers fly out the window as soon as your math involves any kind of more complicated trigonometry, or even a square root…

You can turn them back into rationals, (rational (sqrt 2d0)) => 6369051672525773/4503599627370496

Or write your own operations that compute to the precision you want.

Re: Beware of Fast-Math

#35

Post by wffurr »

rlpb wrote: post #9

> I mean, the whole point of fast-math is trading off speed with correctness. If fast-math was to give always the correct results, it wouldn’t be fast-math, it would be the standard way of doing math. A similar warning applies to -O3. If an optimization in -O3 were to reliably always give better results, it wouldn't be in -O3; it'd be in -O2. So blindly compiling with -O3 also doesn't seem like a great idea.

If the answer can be wrong, you can make it as fast as you want.

Re: Beware of Fast-Math

#36

Post by Affric »

emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

How does IEEE 754 prevent auto-vectorisation?

Re: Beware of Fast-Math

#37

Post by cycomanic »

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boost.

I particularly find the discussion of - fassociative-math because I assume that most writers of some code that translates a mathetical formula to into simulations will not know which would be the most accurate order of operations and will simply codify their derivation of the equation to be simulated (which could have operations in any order). So if this switch changes your results it probably means that you should have a long hard look at the equations you're simulating and which ordering will give you the most correct results.

That said I appreciate that the considerations might be quite different for libraries and in particular simulations for mathematics.

Re: Beware of Fast-Math

#38

Post by Kubuxu »

Affric wrote: post #36
emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

How does IEEE 754 prevent auto-vectorisation?

IIRC reordering additions can cause the result to change which makes auto-vectorisation tricky.

Re: Beware of Fast-Math

#39

Post by [deleted] »

nextaccountic wrote: post #20
eqvinox wrote: post #19

Earlier quoted context omitted.

Ah sorry I misunderstood and thought this API was for the other way around, i.e. forbidding "unsafe" operations. (I guess the question reverses to setting those flags) ('Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic.' :D)

> ('Naming: "algebraic" is not very descriptive of what this does since the operations themselves are algebraic.' :D) Okay, the floating point operations are literally algebraic (they form an algebra) but they don't follow some common algebraic properties like associativity. The linked tracking issue itself acknowledges that: > Naming: "algebraic" is not very descriptive of what this does since the operations themsel…

[deleted]

Re: Beware of Fast-Math

#40

Post by zokier »

Affric wrote: post #24

For non-associativity what is the best way to order operations? Is there an optimal order for precision whereby more similar values are added/multiplied first? EDIT: I am now reading Goldberg 1991 Double edit: Kahan Summation formula. Goldberg is always worth going back to.

Herbie can optimize arbitrary floating point expressions for accuracy

https://herbie.uwplse.org/

Post reply on HN