Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

181–190 of 233 posts

Re: Beware of Fast-Math

#181

Post by AlotOfReading »

on_the_train wrote: post #52
cycomanic wrote: post #37

I think this article overstates the importance of the problems even for scientific software. In the scientific code I've written, noise processes are often orders of magnitude larger than what what is discussed here and I believe this applies to many (most?) simulations modelling the real world (i.e. Physics chemistry,..). At the same time enabling fast-math has often yielded a very significant (>10%) performance boo…

I worked in cad, robotics and now semiconductor optics. In every single field, floating precision down to the very last digits was a huge issue

"precision" is an ambiguous term here. There's reproducibility (getting the same results every time), accuracy (getting as close as possible to same results computed with infinite precision), and the native format precision.

ffast-math is sacrificing both the first and the second for performance. Compilers usually sacrifice the first for the second by default with things like automation fma contraction. This isn't a necessary trade-off, it's just easier.

There's very few cases where you actually need accuracy down to the ULP though. No robot can do anything meaningful with femtometer+ precision, for example. Instead you choose a development balance between reproducibility (relatively easy) and accuracy (extremely hard). In robotics, that will usually swing a bit towards reproducibility. CAD would swing more towards accuracy.

Re: Beware of Fast-Math

#182

Post by kstrauser »

sfn42 wrote: post #135
londons_explore wrote: post #43

Earlier quoted context omitted.

It would be nice if there was some syntax for "math order matters, this is the order I want it done in". Then all other math will be fast-math, except where annotated.

I thought most languages have this? If you simply write a formula operations are ordered according to the language specifiction. If you want different ordering you use parentheses. Not sure how that interacts with this fast math thing, I don't use C

That’s a different kind of ordering.

Imagine a function like Python’s `sum(list)`. In abstract, Python should be able to add those values in any order it wants. Maybe it could spawn a thread so that one process sums the first half in the list, another sums the second half at the same time, and then you return the sum of those intermediate values. You could imagine a clever `sum()` being many times faster, especially using SIMD instructions or a GPU or something.

But alas, you can’t optimize like that with common IEEE-754 floats and expect to get the same answer out as when using the simple one-at-a-time addition. The result depends on what order you add the numbers together. Order them differently and you very well may get a different answer.

That’s the kind of ordering we’re talking about here.

Re: Beware of Fast-Math

#183

Post by Measter »

Sharlin wrote: post #27
eqvinox wrote: post #23

Earlier quoted context omitted.

It doesn't feel appropriate to comment there for me not knowing any Rust really, but "lax_" (or "relax_") would have the extra benefit of being very short. (Is this going to overload operators or are people going to have to type this… a lot… ?)

Rust has some precedence for adding convenience newtypes with overloaded operators (eg. `Wrapping ´ for `I.wrapping_add(I)` etc). Such a wrapper isn't currently proposed AFAIK but there's no reason one couldn't be added in the future I believe.

For giggles, here's one I whipped up, along with an example use: https://godbolt.org/z/Eezj35dzc

Re: Beware of Fast-Math

#184

Post by scott_w »

Filligree wrote: post #141
scott_w wrote: post #96

Earlier quoted context omitted.

Fixed point Decimal is your friend here. I’m guessing you buy tokens in increments of 1,000,000 so it isn’t too much of an issue to account for. You can then normalise in your accounting so 1,000,000 is just “1 unit,” or you can just account in increments of 1,000,000 but that does start looking weird (but might be necessary!)

No, billing happens per-token. It’s entirely necessary to use billionths of a dollar here, if you don’t use floating point.

In which case, I’d look at this thread https://news.ycombinator.com/item?id=44145263

Re: Beware of Fast-Math

#185

Post by 01HNNWZ0MV43FF »

gamescr wrote: post #137
usefulcat wrote: post #122

Earlier quoted context omitted.

You can certainly make trading systems that work using floating point, but there are just so many fewer edge cases to consider when using fixed point. With fixed point and at least 2 decimal places, 10.01 + 0.01 is always exactly equal to 10.02. But with FP you may end up with something like 10.0199999999, and then you have to be extra careful anywhere you convert that to a string that it doesn't get truncated to 10.…

I work on game engines and the problem with floats isn't on small values like 10.01 but on large ones like 400,010.01 that's when the precision wildly varies.

Lua is telling me 0.1 + 0.1 == 0.2, but 0.1 + 0.2 != 0.3. That's 64-bit precision. The issue is not with precision, but with 1/10th being a repeating decimal in binary.

Re: Beware of Fast-Math

#186

Post by [deleted] »

razighter777 wrote: post #65

The worst thing that strikes fear into me is seeing floating points used for real world currency. Dear god. So many things can go wrong. I always use unsigned integers counting number of cents. And if I gotta handle multiple currencies, then I'll use or make a wrapper class.

[deleted]

Re: Beware of Fast-Math

#187

Post by cstrahan »

adgjlsfhk1 wrote: post #110
dataangel wrote: post #91

Earlier quoted context omitted.

you can't accurately represent 10 cents with floats, 0.1 is not directly representable. same with 1 cent, 0.01. Seems like if you do and significant math on prices you should run into rounding issues pretty quickly?

no. Float64 has 16 digits of precision. Therefore even if you're dealing with trillions of dollars, you have accuracy down to the thousandth of a cent.

You might want to re-study this topic.

The decimal number 0.1 has an infinitely repeating binary fraction.

Consider how 1/3 in decimal is 0.33333… If you truncate that to some finite prefix, you no longer have 1/3. Now let’s suppose we know, in some context, that we’ll only ever have a finite number of digits — let’s say 5 digits after the decimal point. Then, if someone asks “what fraction is equivalent to 0.33333?”, then it is reasonable to reply with “1/3”. That might sound like we’re lying, but remember that we agreed that, in this context of discussion, we have a finite number of digits — so the value 1/3 outside of this context has no way of being represented faithfully inside this context, so we can only assume that the person is asking about the nearest approximation of “1/3 as it means outside this context”. If the person asking feels lied to, that’s on them for not keeping the base assumptions straight.

So back to floating point, and the case of 0.1 represented as 64 bit floating point number. In base 2, the decimal number 0.1 looks like 0.0001100110011… (the 0011 being repeated infinitely). But we don’t have an infinite number of digits. The finite truncation of that is the closest we can get to the decimal number 0.1, and by the same rationale as earlier (where I said that equating 1/3 with 0.33333 is reasonable), your programming language will likely parse “0.1” as a f64 and print it back out as such. However, if you try something like (a=0.1; a+a+a) you’ll likely be surprised at what you find.

Re: Beware of Fast-Math

#188

Post by CamouflagedKiwi »

rlpb wrote: post #22
CamouflagedKiwi wrote: post #15

Earlier quoted context omitted.

The optimisations in -O3 aren't supposed to give incorrect results. They're not in -O2 because they make a more aggressive space/speed tradeoff or increase compile times more significantly. In the same way, the optimisations in -O2 are not meant to be less correct than -O1, but they aren't in that group for similar reasons. -Ofast is the 'dangerous' one. (It includes -ffast-math).

> The optimisations in -O3 aren't supposed to give incorrect results. I didn't mean to imply that they result in incorrect results. > they make a more aggressive space/speed tradeoff... Right...so "better" becomes subjective, depends on the use case, so it doesn't make sense to choose -O3 blindly unless you understand the trade-offs and want that side of them for the particular builds you're doing. Things that everyo…

If they're things that everyone wants, why aren't they in -O1?

Re: Beware of Fast-Math

#189

Post by neepi »

kvemkon wrote: post #101
teleforce wrote: post #62

“Nothing brings fear to my heart more than a floating point number.” - Gerald Jay Sussman Is there any IEEE standards committee working on FP alternative for examples Unum and Posit [1],[2]. [1] Unum & Posit: https://posithub.org/about [2] The End of Error: https://www.oreilly.com/library/view/the-end-of/978148223986...

I'm wondering, why there are still no announcements for hardware support of such approaches in CPUs.

HP had proper deterministic decimal arithmetic since the 1970s.

Re: Beware of Fast-Math

#190

Post by einpoklum »

stingraycharles wrote: post #48
kzrdude wrote: post #41

Earlier quoted context omitted.

If you write a loop `for x in array { sum += x }` Then your program is a specification that you want to add the elements in exactly that order, one by one. Vectorization would change the order.

Yup, because of the imprecision of floating points, cannot just assume that “(a + c) + (b + d)” is the same as “a + b + c + d”. It would be pretty ironic if at some point fixed point / bignum implementations end up being faster because of this.

I wonder... couldn't there just be some library type for this, e.g. `associative::float` and `associative::doube` and such (in C++ terms), so that compilers can ignore non-associativity for actions on values of these types? Or attributes one can place on variables to force assumption of associativity?
Post reply on HN