Practically Accurate Floating-Point Math
computer.org
Practically Accurate Floating-Point Math
1–10 of 33 posts
Re: Practically Accurate Floating-Point Math
#2This article (like all other IEEE Computing Now articles) is free to download for a limited time.
Re: Practically Accurate Floating-Point Math
#3If you are interested in writing accurate numerical algorithms, as further reading I highly recommend reading "Accuracy and Stability of Numerical Algorithms" by Nick Higham, which is a very good comprehensive book. Like this article says, floating-point arithmetic is often thought of as mysterious, but it really isn't: it just obeys its own precisely specified rules. I think it's very good to dispel the mystery, so I really like this article.
Another way to do the same type of error analysis of numerical algorithms is to use a parameter sensitivity library. In the end, numerical stability can be thought of as the sensitivity of the final answer to separate independent changes in each intermediate value obtained in the computation of the answer. This is sometimes overlooked, I think, and it's quite easy to implement as well.
I have one quibble. They say "Use (x-y)(x+y) instead of x^2-y^2". This is true, but (in my opinion) misleading: the function (x,y) -> x^2-y^2 is itself ill-conditioned when x is near y, and using a more numerically stable algorithm to evaluate an ill-conditioned function will not improve the function's condition. If x and y are close to each other, the inaccuracy introduced by using x^2-y^2 is close to the inaccuracy due to using approximate values for x and y. So it you have x^2-y^2, I think the first thing you should worry about is whether your computation is ill-conditioned, and only then worry whether it's numerically stable. I say this because I sometimes see people try to evaluate x^2-1, notice it is inaccurate for x near 1, and try to replace x^2-1 with (x-1)*(x+1), which is still inaccurate.
Re: Practically Accurate Floating-Point Math
#4Even though refreshing and enlightening, the article doesn't cover another (major) reason why floating-point math is generally avoided (especially in high-performance applications): computational slowdown when dealing with subnormals[1][2].
I feel that there is a lot of overlap between cases where you may want to minimize error while at the same time still be performant (simulations, ray tracing, rendering, etc.). So you're left with a situation where you end up minimizing error, but still being slow.
Re: Practically Accurate Floating-Point Math
#5Even though refreshing and enlightening, the article doesn't cover another (major) reason why floating-point math is generally avoided (especially in high-performance applications): computational slowdown when dealing with subnormals[1][2]. I feel that there is a lot of overlap between cases where you may want to minimize error while at the same time still be performant (simulations, ray tracing, rendering, etc.). So…
A couple points:
Support for subnormals can easily be disabled (usually be enabling "Flush to Zero" behavior) to allow for better performance on essentially all recent hardware.
Over the last few years, subnormal stalls have shrunk rapidly, and gone away entirely in some cases. Intel started this process with the Sandybridge architecture. These stalls are also greatly reduced in some arm64 implementations.
Re: Practically Accurate Floating-Point Math
#6If you are interested in writing accurate numerical algorithms, as further reading I highly recommend reading "Accuracy and Stability of Numerical Algorithms" by Nick Higham, which is a very good comprehensive book. Like this article says, floating-point arithmetic is often thought of as mysterious, but it really isn't: it just obeys its own precisely specified rules. I think it's very good to dispel the mystery, so…
The correct way to evaluate x^2 - 1 is by using fma(x,x,-1). Now that Intel and AMD have finally made FMA available in hardware on commodity parts (better late than never!), it's realistic to start using this much more freely.
Re: Practically Accurate Floating-Point Math
#7This article (like all other IEEE Computing Now articles) is free to download for a limited time.
Is it me, or have all the ells (the letter L) been replaced by ones (the digit 1) in the PDF?
Edit: It's me. It's just a font with sloping serifs, which is what confused me.
Re: Practically Accurate Floating-Point Math
#8If you are interested in writing accurate numerical algorithms, as further reading I highly recommend reading "Accuracy and Stability of Numerical Algorithms" by Nick Higham, which is a very good comprehensive book. Like this article says, floating-point arithmetic is often thought of as mysterious, but it really isn't: it just obeys its own precisely specified rules. I think it's very good to dispel the mystery, so…
The correct way to evaluate x^2 - 1 is by using fma(x,x,-1). Now that Intel and AMD have finally made FMA available in hardware on commodity parts (better late than never!), it's realistic to start using this much more freely.
The expression won't turn to be good conditioned only by using fma. You actually missed the point of conistonwater's post.
Re: Practically Accurate Floating-Point Math
#9If you are interested in writing accurate numerical algorithms, as further reading I highly recommend reading "Accuracy and Stability of Numerical Algorithms" by Nick Higham, which is a very good comprehensive book. Like this article says, floating-point arithmetic is often thought of as mysterious, but it really isn't: it just obeys its own precisely specified rules. I think it's very good to dispel the mystery, so…
The correct way to evaluate x^2 - 1 is by using fma(x,x,-1). Now that Intel and AMD have finally made FMA available in hardware on commodity parts (better late than never!), it's realistic to start using this much more freely.
That won't do much for x near 1, because the function x^2-1 is itself ill-conditioned. In other words, it is relevant that the floating-point value of x is itself only an approximation to some true value of x. So computing x^2-1 exactly for a given floating-point value of x does not give a good approximation to the true value of x^2-1. This is a mathematical property of the function x^2-1, and cannot be fixed with any algorithm. This is basically why I consider the example x^2-1 => (x-y)(x+y) misleading.
Re: Practically Accurate Floating-Point Math
#10Earlier quoted context omitted.
The correct way to evaluate x^2 - 1 is by using fma(x,x,-1). Now that Intel and AMD have finally made FMA available in hardware on commodity parts (better late than never!), it's realistic to start using this much more freely.
That won't do much for x near 1, because the function x^2-1 is itself ill-conditioned. In other words, it is relevant that the floating-point value of x is itself only an approximation to some true value of x. So computing x^2-1 exactly for a given floating-point value of x does not give a good approximation to the true value of x^2-1. This is a mathematical property of the function x^2-1, and cannot be fixed with an…
There are multiple ways to analyze computations; condition number is one of them. It is relevant when inputs are assumed to be approximations to some hidden "correct" value. However, this assumption is not always warranted when dealing with floating-point numbers; sometimes, we would instead like to analyze errors under the assumption that the inputs are exact values. When this is the case, fma(x,x,-1) produces a correctly rounded result, whereas (x-1)(x+1) does not, and the ill-conditioning of the function x^2 - 1 does not enter into the analysis anywhere.
If you are used to working with complex computations (most of what is usually referred to as numerical analysis, for example), you likely do not find yourself in the latter situation very often. However, for those of us who work on low-level details of floating-point on a regular basis, the latter analysis is often critical.