Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Beware of fast-math

simonbyrne.github.io

Post reply on HN
Print view View original

31–40 of 111 posts

Re: Beware of fast-math

#31

Post by Someone »

adrian_b wrote: post #28
SeanLuke wrote: post #6

The other examples he gave trade off significant math deficiencies for small speed gains. But flushing subnormals to zero can produce a MASSIVE speed gain. Like 1000x. And including subnormals isn't necessarily good floating point practice -- they were rather controversial during the development of IEEE 754 as I understand it. The tradeoff here is markedly different than in the other cases.

Flushing subnormals to zero produces speed gains only on certain CPU models, while on others it almost does not have any effect. For example Zen CPUs have negligible penalties for handling denormals, but many Intel models have a penalty between 100 and 200 clock cycles for an operation with denormals. Even on the CPU models with slow denormal processing, a speedup between 100 and 1000 exists only for the operation wi…

> The only controversy that has ever existed about denormals is that handling them at full speed increases the cost of the FPU, so lazy or greedy companies, i.e. mainly Intel, have preferred to add the flush-to-zero option for gamers

You could also say some companies have been kind enough to make hardware for gamers that doesn’t have costly features they do not need.

Re: Beware of fast-math

#32

Post by dahart »

adgjlsfhk1 wrote: post #30
dahart wrote: post #29

Earlier quoted context omitted.

> which suggests that there ARE ways of getting FMAs, without the sloppiness of fast-math. There are ways, indeed, but they are pretty slow, it’s prioritizing accuracy over performance. And they’re still pretty tricky too. The most practical alternative for float FMA might be to use doubles, and for double precision FMA might be to bump to a 128 bit representation. Here’s a paper on what it takes to do FMA emulation:…

That's not what the parent meant. The parent meant that there are ways of generating fma instructions without using fast-math. Emulating an fma instruction is almost always a bad idea (I should know I've written fma-emulation before. It sucks)

Oh, my mistake, thanks. Yes you can use FMA instructions without the fast-math compiler flag for sure. Emulation being a bad idea is the impression I got; I’m glad to hear the confirmation from experience.

Re: Beware of fast-math

#33

Post by simonbyrne »

pavpanchekha wrote: post #27

Fun fact: when working on Herbie ( http://herbie.uwplse.org ), our automated tool for reducing floating-point error by rearranging your mathematical expressions, we found that fast-math often undid Herbie's improvements. In a sense, Herbie and fast-math are opposites: one makes code more accurate (sometimes slower, sometimes faster), while the other makes code faster (sometimes less accurate, sometimes more).

Herbie is a great tool, especially for teaching.

Re: Beware of fast-math

#34

Post by simonbyrne »

zoomablemind wrote: post #14

On the subject of the floating-point math in general, I wonder what's the practical way to treat the extreme order values (close to zero ~ 1E-200, or infinity ~ 1E200, but not zero or inf)? This can take place in some iterative methods, expansion series, or around some singularities. How reliable is it to keep the exreme orders in expectation that the resp. quatities would cancel the orders properly yielding a meanin…

The key thing about floating point is that it maintains relative accuracy: in your case, if you have say f(x) and g(x) are both O(1e200), and are correct to some small relative tolerance, say 1e-10 (that is, the absolute error is 1e190). Then the relative for f(x)/g(x) stays nicely bounded to about 2e-10.

However if you do f(x) - g(x), the absolute error is on the order of 2e190: if f(x) - g(x) is small, then now the relative error can be huge (this is known as catastrophic cancellation).

Re: Beware of fast-math

#35

Post by named-user »

pavpanchekha wrote: post #27

Fun fact: when working on Herbie ( http://herbie.uwplse.org ), our automated tool for reducing floating-point error by rearranging your mathematical expressions, we found that fast-math often undid Herbie's improvements. In a sense, Herbie and fast-math are opposites: one makes code more accurate (sometimes slower, sometimes faster), while the other makes code faster (sometimes less accurate, sometimes more).

The docs for fast math literally say this "This option is not turned on by any -O option besides -Ofast since it can result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions. It may, however, yield faster code for programs that do not require the guarantees of these specifications."

So what did you expect?

I was taught in my undergrad class not to use fast math if you ever wanted accuracy. I don't understand how you missed this?

Re: Beware of fast-math

#36

Post by dlsa »

Never considered fast-math. I get the sense its useful but can create awkward and/or unexpected surprises. If I was to use it I'd have to have a verification test harness as part of some pipeline to comfirm no weirdness. Literally a bunch of example canary calculations to determine if fast-math will kill or harm some real use case.

Is this a sensible approach? What are others experiences around this? I've never bothered with this kind of optimisation and I now vaguely feel like I'm missing out.

I tend to use calculations for deterministic purposes rather than pure accuracy. 1+1=2.1 where the answer is stable and approximate is still better and more useful than 1+1=2.0 but where the answer is unstable. Eg because one of those is 0.9999999 and the precision triggers some edge case.

Re: Beware of fast-math

#37

Post by an1sotropy »

dahart wrote: post #29
an1sotropy wrote: post #8

Earlier quoted context omitted.

(in case anyone reading doesn't know: FMA = Fused Multiply and Add, as in a*b+c, an operation on 3 values, which increases precision by incurring rounding error once instead of twice) I'm not an expert on this, but for my own code I've been meaning to better understand the discussion here [1], which suggests that there ARE ways of getting FMAs, without the sloppiness of fast-math. [1] https://stackoverflow.com/questi…

> which suggests that there ARE ways of getting FMAs, without the sloppiness of fast-math. There are ways, indeed, but they are pretty slow, it’s prioritizing accuracy over performance. And they’re still pretty tricky too. The most practical alternative for float FMA might be to use doubles, and for double precision FMA might be to bump to a 128 bit representation. Here’s a paper on what it takes to do FMA emulation:…

I remember a teacher who said (when I was a student) something like "if you care about precision use double". Now that I'm teaching, I force students to only use single-precision "float"s in their code, with the message that FP precision is a finite resource, and you don't learn how to manage any resource by increasing its supply. I think my students hate me.

Re: Beware of fast-math

#38

Post by pavpanchekha »

named-user wrote: post #35
pavpanchekha wrote: post #27

Fun fact: when working on Herbie ( http://herbie.uwplse.org ), our automated tool for reducing floating-point error by rearranging your mathematical expressions, we found that fast-math often undid Herbie's improvements. In a sense, Herbie and fast-math are opposites: one makes code more accurate (sometimes slower, sometimes faster), while the other makes code faster (sometimes less accurate, sometimes more).

The docs for fast math literally say this "This option is not turned on by any -O option besides -Ofast since it can result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions. It may, however, yield faster code for programs that do not require the guarantees of these specifications." So what did you expect? I was taught in my undergrad class…

Hi, I noticed you're a new user, so in that spirit this is not the kind of comment that is well-received on HN. It reads as arrogant and condescending, though I assume that was not your intention. Since I publish regularly in this area, the condescension feels especially off-base.

To answer the question in good faith: while the documentation clearly states that fast-math can reduce accuracy, in isolation the accuracy changes are often small, and moreover in isolation the accuracy changes often look positive. So I was surprised, when evaluating Herbie, that it frequently caused large, negative changes in accuracy, specifically by undoing changes made to improve accuracy.

Re: Beware of fast-math

#39

Post by wiml »

adgjlsfhk1 wrote: post #5
kzrdude wrote: post #4

It looks like -fassociative-math is "safe" in the sense that it can not be used to get UB in working code? That's a good property to make it easier to use in the right context.

Yeah. It's safe in that you won't get UB, but it's bad in that you can get arbitrarily wrong answers.

To be fair, if you're using floating point at all you can get arbitrarily wrong answers. The nice thing about ieee754 conformance is that you can, with a lot of expertise, somewhat reason about the kinds of error you're getting. But for code that wasn't written by someone skilled in numerical techniques, and that's the vast majority of fp code, is fast-math worse than the status quo?

Re: Beware of fast-math

#40

Post by stephencanon »

StefanKarpinski wrote: post #13
mbauman wrote: post #9

Earlier quoted context omitted.

See the one footnote: you can re-associate a list of 2046 numbers such that they sum to _any_ floating point number between 0 and 2^970. https://discourse.julialang.org/t/array-ordering-and-naive-s...

To be fair though, as noted further down in that thread, naive left-to-right summation is the worst case here since the tree of operations is as skewed as possible. I think that any other tree shape is better and compilers will tend to make the tree more balanced, not less, when they use associativity to enable SIMD optimizations. So while reassociation is theoretically arbitrarily bad, in practice it's probably most…

This is exactly right. Also, _all_ of the summation orderings satisfy the usual backwards-stability bounds, and so all of them are perfectly stable in normal computations.
Post reply on HN