Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of Fast-Math

simonbyrne.github.io

Post reply on HN
Print view View original

211–220 of 233 posts

Re: Beware of Fast-Math

#211

Post by adrian_b »

dzaima wrote: post #176
emn13 wrote: post #31

I get the feeling that the real problem here are the IEEE specs themselves. They include a huge bunch of restrictions that each individually aren't relevant to something like 99.9% of floating point code, and probably even in aggregate not a single one is relevant to a large majority of code segments out in the wild. That doesn't mean they're not important - but some of these features should have been locally opt-in,…

The precise requirements of IEEE-754 may not be important for any given program, but as long as you want your numbers to have any form of well-defined semantics beyond "numbers exist, and here's a list of functions that do Something™ that may or may not be related to their name", any number format that's capable of (approximately) storing both 10^20 and 10^-20 in 64 bits is gonna have those drawbacks. AFAIK GPU code…

A CUDA "kernel" is the same thing as what has been called "parallel DO" or "parallel FOR" since 1963, or perhaps even earlier.

This is slightly obfuscated by not using a keyword like "for" or "do", by the fact that the body of the loop (the "kernel") is written in one place and and the header of the loop (which gives the ranges for the loop indices) is written in another place, and by the fact that the loop indices have standard names.

A "parallel for" may have as well a syntax identical with a sequential "for". The difference is that for the "parallel for" the compiler knows that the iterations are independent, so they may be scheduled to be executed concurrently.

NVIDIA has been always greatly annoying by inventing a huge amount of new terms that are just new words for old terms that have been used for decades in the computing literature, with no apparent purpose except of obfuscating how their GPUs really work. Worse, AMD has imitated NVIDIA, by inventing their own terms that correspond to those used by NVIDIA, but they are once again different.

Re: Beware of Fast-Math

#212

Post by Dylan16807 »

anthk wrote: post #85
vanderZwan wrote: post #46

Earlier quoted context omitted.

> These tell the compiler to assume for optimization purposes that floats are associative and so on (ie. algebraic), even when in reality they aren't. I wonder if it is possible to add an additional constraint that guarantees the transformation has equal or fewer numerical rounding errors. E.g. for floating point doubles (0.2 + 0.1) - 0.1 results in 0.20000000000000004, so I would expect that transforming some (A + B…

Under EForth with FP done in software: 2 f 1 f 1 f f+ f- f. 0.000 ok PFE, I think reusing the GLIBC math library: 2e0 1e0 1e0 f+ f- f. 0.000000 ok

Every implementation of floating point can handle small to medium integers without rounding. So that example doesn't show anything we can learn from.

Re: Beware of Fast-Math

#213

Post by vanderZwan »

kvemkon wrote: post #101
teleforce wrote: post #62

“Nothing brings fear to my heart more than a floating point number.” - Gerald Jay Sussman Is there any IEEE standards committee working on FP alternative for examples Unum and Posit [1],[2]. [1] Unum & Posit: https://posithub.org/about [2] The End of Error: https://www.oreilly.com/library/view/the-end-of/978148223986...

I'm wondering, why there are still no announcements for hardware support of such approaches in CPUs.

Gustavson's last presentation starts with him holding an actual piece of hardware supporting posits, fwiw.

https://m.youtube.com/watch?v=vzVlQhaAZtQ

Re: Beware of Fast-Math

#214

Post by anthk »

Dylan16807 wrote: post #212
anthk wrote: post #85

Earlier quoted context omitted.

Under EForth with FP done in software: 2 f 1 f 1 f f+ f- f. 0.000 ok PFE, I think reusing the GLIBC math library: 2e0 1e0 1e0 f+ f- f. 0.000000 ok

Every implementation of floating point can handle small to medium integers without rounding. So that example doesn't show anything we can learn from.

Buth for Forth you can have a totally working software floating point made from the people whose later would set the IEEE standard. Think about very small microcontrollers without FP in hardware, you can be sure that a software FP implementation will perfectly work under the standard thresolds.

Re: Beware of Fast-Math

#215

Post by Measter »

eqvinox wrote: post #209
Measter wrote: post #183

Earlier quoted context omitted.

For giggles, here's one I whipped up, along with an example use: https://godbolt.org/z/Eezj35dzc

Having done unspeakable things with the C preprocessor myself, this Rust macro soup truly warms my heart <3

Oh this is simple, this is just a bit of basic expansion to generate boilerplate. The real experts can do some gnarly and amazing stuff with macros.

Re: Beware of Fast-Math

#216

Post by anthk »

adrian_b wrote: post #211
dzaima wrote: post #176

Earlier quoted context omitted.

The precise requirements of IEEE-754 may not be important for any given program, but as long as you want your numbers to have any form of well-defined semantics beyond "numbers exist, and here's a list of functions that do Something™ that may or may not be related to their name", any number format that's capable of (approximately) storing both 10^20 and 10^-20 in 64 bits is gonna have those drawbacks. AFAIK GPU code…

A CUDA "kernel" is the same thing as what has been called "parallel DO" or "parallel FOR" since 1963, or perhaps even earlier. This is slightly obfuscated by not using a keyword like "for" or "do", by the fact that the body of the loop (the "kernel") is written in one place and and the header of the loop (which gives the ranges for the loop indices) is written in another place, and by the fact that the loop indices h…

xargs does a parallel for too. And OFC Forth people might did that too in a breeze.

Re: Beware of Fast-Math

#217

Post by adgjlsfhk1 »

layer8 wrote: post #202
adgjlsfhk1 wrote: post #192

Earlier quoted context omitted.

having multiple NaNs and no spec for how they should behave feels like such an unforced error to me

For mathematical use, NaN payloads shouldn’t matter, and behave identically (aside from quiet vs. signaling NaNs). It also doesn’t matter for equality comparison, because NaNs always compare unequal.

from the user perspective it's not too bad, but from the compiler perspective it is. The result of this is that LLVM has decided that trying to figure out which nan you got (e.g. by casting to an Int and comparing) is UB, which means pretty much every floating point operation becomes non-deterministic.

This also adds extra complexity to the CPU. you need special hardware for == rather than just using the perfectly good integer unit, and every fpu operation needs to devote a bunch of transistors to handling this nonsense that buys the user absolutely nothing.

there are definitely things to criticize about the design of Posits, but the thing they 100% get right is having a single NaN and sane ordering semantics

Re: Beware of Fast-Math

#218

Post by kbolino »

nurettin wrote: post #208
kbolino wrote: post #207

Earlier quoted context omitted.

You can round to the nearest .0078125 (1/128) but not to the nearest .01 (1/100) because that number cannot be represented in float64. To round to the nearest cent, you would need to make cents your units (i.e. the quantity "1 dollar" would be represented as 100 instead of 1.0).

Within the general context of this discussion, "cannot be represented" is a red herring. You don't need to have a representation of the exact number 0.1 if you can tolerate errors after the 7th decimal (and it turns out you can). And 0.1+0.1+0.1 does not have to be comparable with 0.3 using operator==. You have an is_close function for that. And accumulation is not an issue because you have rounding and fma for that.

Ok, but in the context of this thread, it is important. Remember that I'm replying to the statement "For added assurance, just round to the nearest cent after each operation." That is misleading advice and the behavior of floats is just context for why.

First of all, a lot of languages don't include arbitrary rounding in their math libraries at all, only having rounding to integers. Second, in the docs of Python, which does have arbitrary rounding, it specifically says:

    Note: The behavior of round() for floats can be surprising: for example, round(2.675, 2) gives 2.67 instead of the expected 2.68. [...]
Thus I think what I said stands: you cannot round to nearest cent reliably all the time, assuming cent means 0.01. The only rounding you can sort of trust is display rounding because it actually happens after converting to base 10. It's why 2.675 will print as 2.675 in Python even though it won't round as you'd expect. But you'd only do that once at the end of a chain of operations.

In a lot of cases, errors like these don't matter, but the key point is that if the errors don't matter, then they don't need to be "assured" away by dubious rounding either.

Re: Beware of Fast-Math

#219

Post by adrian_b »

anthk wrote: post #216
adrian_b wrote: post #211

Earlier quoted context omitted.

A CUDA "kernel" is the same thing as what has been called "parallel DO" or "parallel FOR" since 1963, or perhaps even earlier. This is slightly obfuscated by not using a keyword like "for" or "do", by the fact that the body of the loop (the "kernel") is written in one place and and the header of the loop (which gives the ranges for the loop indices) is written in another place, and by the fact that the loop indices h…

xargs does a parallel for too. And OFC Forth people might did that too in a breeze.

That's right and the same is done by the improved version of xargs, GNU "parallel".

Re: Beware of Fast-Math

#220

Post by nurettin »

kbolino wrote: post #218
nurettin wrote: post #208

Earlier quoted context omitted.

Within the general context of this discussion, "cannot be represented" is a red herring. You don't need to have a representation of the exact number 0.1 if you can tolerate errors after the 7th decimal (and it turns out you can). And 0.1+0.1+0.1 does not have to be comparable with 0.3 using operator==. You have an is_close function for that. And accumulation is not an issue because you have rounding and fma for that.

Ok, but in the context of this thread, it is important. Remember that I'm replying to the statement "For added assurance, just round to the nearest cent after each operation." That is misleading advice and the behavior of floats is just context for why. First of all, a lot of languages don't include arbitrary rounding in their math libraries at all , only having rounding to integers. Second, in the docs of Python, wh…

floats are also a red herring. Maybe you can continue with someone else.
Post reply on HN