Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of fast-math

simonbyrne.github.io

Post reply on HN
Print view View original

41–50 of 111 posts

Re: Beware of fast-math

#41

Post by zoomablemind »

simonbyrne wrote: post #34
zoomablemind wrote: post #14

On the subject of the floating-point math in general, I wonder what's the practical way to treat the extreme order values (close to zero ~ 1E-200, or infinity ~ 1E200, but not zero or inf)? This can take place in some iterative methods, expansion series, or around some singularities. How reliable is it to keep the exreme orders in expectation that the resp. quatities would cancel the orders properly yielding a meanin…

The key thing about floating point is that it maintains relative accuracy: in your case, if you have say f(x) and g(x) are both O(1e200), and are correct to some small relative tolerance, say 1e-10 (that is, the absolute error is 1e190). Then the relative for f(x)/g(x) stays nicely bounded to about 2e-10. However if you do f(x) - g(x), the absolute error is on the order of 2e190: if f(x) - g(x) is small, then now the…

Both f(x) and g(x) could be calc'ed to proper machine precision (say, GSL double prec ~ 1E-15). Would this imply, that beyond the machine precision the values are effectively fixed to resp. zero or infinity, instead of carrying around the extreme orders of magnitude?

Re: Beware of fast-math

#42

Post by physicsguy »

mhh__ wrote: post #26
vchuravy wrote: post #2

Especially the fact that loading a library compiled with GCC and fast math on, can modify the global state of the program... It's one of the most baffling decisions made in the name of performance. I would really like for someone to take fast math seriously, and to provide well scoped and granular options to programmers. The Julia `@fastmath` macro gets close, but it is two broad. I want to control the flags individu…

D (LDC) lets you control the flags on a per function basis. So one can (and we do at work) have @optmath which is a specific set of flags (just a value we defined at compile time, the compiler recognizes it as a UDA) we want as opposed to letting the compiler bulldoze everything.

You can do that on a per-compiler basis for e.g. with

#pragma GCC optimize(“fast-math")

Re: Beware of fast-math

#43

Post by simonbyrne »

zoomablemind wrote: post #41
simonbyrne wrote: post #34

Earlier quoted context omitted.

The key thing about floating point is that it maintains relative accuracy: in your case, if you have say f(x) and g(x) are both O(1e200), and are correct to some small relative tolerance, say 1e-10 (that is, the absolute error is 1e190). Then the relative for f(x)/g(x) stays nicely bounded to about 2e-10. However if you do f(x) - g(x), the absolute error is on the order of 2e190: if f(x) - g(x) is small, then now the…

Both f(x) and g(x) could be calc'ed to proper machine precision (say, GSL double prec ~ 1E-15). Would this imply, that beyond the machine precision the values are effectively fixed to resp. zero or infinity, instead of carrying around the extreme orders of magnitude?

I'm not exactly sure what you're asking here, but the point is that "to machine precision" is relative: if f(x) and g(x) are O(1e200), then the absolute error of each is still O(1e185). However f(x)/g(x) will still be very accurate (with absolute error O(1e-15)).

Re: Beware of fast-math

#44

Post by simonbyrne »

dlsa wrote: post #36

Never considered fast-math. I get the sense its useful but can create awkward and/or unexpected surprises. If I was to use it I'd have to have a verification test harness as part of some pipeline to comfirm no weirdness. Literally a bunch of example canary calculations to determine if fast-math will kill or harm some real use case. Is this a sensible approach? What are others experiences around this? I've never bothe…

I tried to lay out a reasonable path: incrementally test accuracy and performance, and only enable the necessary optimizations to get the desired performance. Good tests will catch the obvious catastrophic cases, but some will inevitably be weird edge cases.

As always, the devil is in the details: you typically can't check exact equality, as e.g. reassociating arithmetic can give slightly different (but not necessarily worse) results. So the challenge is coming up with appropriate measure of determining whether something is wrong.

Re: Beware of fast-math

#45

Post by simonbyrne »

wiml wrote: post #39
adgjlsfhk1 wrote: post #5

Earlier quoted context omitted.

Yeah. It's safe in that you won't get UB, but it's bad in that you can get arbitrarily wrong answers.

To be fair, if you're using floating point at all you can get arbitrarily wrong answers. The nice thing about ieee754 conformance is that you can, with a lot of expertise, somewhat reason about the kinds of error you're getting. But for code that wasn't written by someone skilled in numerical techniques, and that's the vast majority of fp code, is fast-math worse than the status quo?

From personal experience, yes: I've seen multiple cases of scientists finding the ultimate cause of their bugs was some fast-math-related optimization.

The problem isn't necessarily the code they wrote themselves: it is often that they've compiled someone else's code or an open source library with fast-math, which broke some internal piece.

Re: Beware of fast-math

#46

Post by willis936 »

I use single precision floating point to save memory and computation in applications where it makes sense. I had a case where I didn't care about the vertical precision of a signal very much. It had a sample rate in the tens of thousands of samples per second. I was generating a sinusoid and transmitting it. On the receiver the signal would become garbled after about a minute. I slapped my head and immediately realized I ran out of precision by using a single precision time value feeding the sin function when t became too large with the small increment.

  sin(single(t)) == bad

  single(sin(t)) == good

Re: Beware of fast-math

#48

Post by adgjlsfhk1 »

willis936 wrote: post #46

I use single precision floating point to save memory and computation in applications where it makes sense. I had a case where I didn't care about the vertical precision of a signal very much. It had a sample rate in the tens of thousands of samples per second. I was generating a sinusoid and transmitting it. On the receiver the signal would become garbled after about a minute. I slapped my head and immediately realiz…

IMO, sinpi(x)=sin(pi*x) is a better function because it does much better here. the regular trig functions are approximately 20% slower for most implementations in order to accurately reduce mod 2pi, while reducing mod 2 is pretty much trivial.

Re: Beware of fast-math

#49

Post by gsteinb88 »

zoomablemind wrote: post #14

On the subject of the floating-point math in general, I wonder what's the practical way to treat the extreme order values (close to zero ~ 1E-200, or infinity ~ 1E200, but not zero or inf)? This can take place in some iterative methods, expansion series, or around some singularities. How reliable is it to keep the exreme orders in expectation that the resp. quatities would cancel the orders properly yielding a meanin…

If you can, working with the logarithms of the intermediate large or small values is one way around the issue

One example talking about this here: http://aosabook.org/en/500L/a-rejection-sampler.html#the-mul...

Post reply on HN