Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Demystifying Floating Point Precision

blog.demofox.org

Post reply on HN
Print view View original

31–40 of 57 posts

Re: Demystifying Floating Point Precision

#31

Post by zmonx »

wruza wrote: post #27
zmonx wrote: post #23

Speaking of floats, I highly recommend John Gustafson's book, The End of Error: Unum Computing : https://www.crcpress.com/The-End-of-Error-Unum-Computing/Gus... It is an amazing compendium of the many ways in which floats fail to give any guarantees in practice. In some cases, due to the format itself, and also because many subtle issues are indicated via internal processor values and flags that are simply not access…

In short, how does u-bit solve multiplication problem when two u-intervals produce >1 interval? Like (1.01+ * 1.01+) = 1.02+ .. 1.04+, in 3-decimal precision

In such cases, you get a ubound that contains the exact value. When the inputs are exact, the result is exact too.

Here is a sample implementation in Julia:

https://github.com/JuliaComputing/Unums.jl

Re: Demystifying Floating Point Precision

#32

Post by pklausler »

slededit wrote: post #18

I've always thought that floats should be accompanied with a precision value specifying the number of significant digits. Ideally updated by hardware in the same operation. All floating point code has subtle bugs if you don't track error accumulation.

It's in a real scientific programming language, viz. Fortran, as SELECTED_REAL_KIND().

Re: Demystifying Floating Point Precision

#33

Post by slededit »

pklausler wrote: post #32
slededit wrote: post #18

I've always thought that floats should be accompanied with a precision value specifying the number of significant digits. Ideally updated by hardware in the same operation. All floating point code has subtle bugs if you don't track error accumulation.

It's in a real scientific programming language, viz. Fortran, as SELECTED_REAL_KIND().

That is the precision of the type, but it does not track accumulated error which quickly gets into the representable range of the type. You need to know both the type, the precision of the data stored in the type and track the operations on it.

Re: Demystifying Floating Point Precision

#34

Post by kraghen »

slededit wrote: post #29
kraghen wrote: post #21

Earlier quoted context omitted.

This sounds like unums, a proposed alternative to IEEE floats, where roughly speaking the significand can have variable size thus only boasting as much precision as the accuracy warrants.

Does it solve the equality issue? An epsilon tracker would allow you to say it's equal within the margin of error for the computation. If so that makes them much more interesting to me. Floating point bugs manifest themselves at non linear parts such as equality and comparison operators.

I haven't really seen the issue of equality dealt with explicitly, but it appears that you can extract the bounds of the interval implied by a given unum. Essentially, it seems like an alternative to interval arithmetic with potentially tighter bounds and faster computation.

Re: Demystifying Floating Point Precision

#35

Post by dragontamer »

colejohnson66 wrote: post #6
dragontamer wrote: post #2

I've stated some notes on Floats before, and I think this blogpost is pretty good overall. The one thing that trips up a lot of people (that wasn't mentioned in this post) is that Floats are non-associative. Try the following in Python (or whatever language that uses Doubles): >>> 1 + 1 + (2. * * 53) > 9007199254740994.0 >>> 1 + (1 + (2. * * 53)) > 9007199254740992.0 claytonjy noted a point of confusion, the above st…

Is there a way to force 80 bit floats if you need them?

Well, it seems odd to me that anybody would want "exactly 80 bit floats".

1. In the case of "doubles", it seems like its more important that your code works the same on all platforms. So you can enable precise IEEE754 behavior, to ensure that your rounding errors and whatnot are the same from platform to platform.

You'll still need to somehow ensure that all of your operations occur in the same order however, IEEE754 only specifies when and how 64-bit numbers get rounded. So sorting your numbers, adding them up from smallest magnitude to largest magnitude is important still.

2. In the case that 64-bits is insufficient precision, you should be using something more precise but also consistent across different platforms.

In case #1: its usually more important that all code "makes the same rounding errors", as opposed to "some code rounds more poorly than other code".

In case #2: you'll want ALL your code to have better rounding behavior. And "Long Double" is still implementation specific. Its probably best to go to a pure software solution (ie: BigNums of some kind) if precision is important to you.

The case #3: some people don't care about the precision, and are willing to have lower precision as long as the results are "nearly correct". Video Game programmers are far more interested in speed for example, so the "fast inverse square root" barely even has 10-bits of precision for example, but the far greater speed wins out in most video game situations.

Re: Demystifying Floating Point Precision

#36

Post by zmonx »

kraghen wrote: post #34
slededit wrote: post #29

Earlier quoted context omitted.

Does it solve the equality issue? An epsilon tracker would allow you to say it's equal within the margin of error for the computation. If so that makes them much more interesting to me. Floating point bugs manifest themselves at non linear parts such as equality and comparison operators.

I haven't really seen the issue of equality dealt with explicitly, but it appears that you can extract the bounds of the interval implied by a given unum. Essentially, it seems like an alternative to interval arithmetic with potentially tighter bounds and faster computation.

Yes, I think these are fair statements.

John Gustafson's presentation also includes a few important points about interval arithmetic:

http://www.johngustafson.net/presentations/Right-SizingPreci...

Re: Demystifying Floating Point Precision

#37

Post by dragontamer »

kraghen wrote: post #14
dragontamer wrote: post #2

I've stated some notes on Floats before, and I think this blogpost is pretty good overall. The one thing that trips up a lot of people (that wasn't mentioned in this post) is that Floats are non-associative. Try the following in Python (or whatever language that uses Doubles): >>> 1 + 1 + (2. * * 53) > 9007199254740994.0 >>> 1 + (1 + (2. * * 53)) > 9007199254740992.0 claytonjy noted a point of confusion, the above st…

It gets even worse: (1 + 2 ^ 53) - 2 ^ 53 evaluates to 0 while 1 + (2 ^ 53 - 2 ^ 53) evaluates to 1 (the correct result), which means that even the operation of adding three floating point numbers has unbounded relative error in the general case. This is a great source of frustration in computational geometry, where this turns from a quantitative issue to a qualitative one, since if we get the sign of an expression w…

> which means that even the operation of adding three floating point numbers has unbounded relative error in the general case.

Its not quite "in the general case".

In "any case with subtraction", there is unbounded relative error. And remember that addition with negative numbers can become subtraction.

Error-management is very important. But if you can GUARANTEE that your operations have the same sign, and that you're only using addition, multiplication, and division... then error is easier to track.

Its that subtraction (aka: cancellation error) that gets ya.

Re: Demystifying Floating Point Precision

#38

Post by zmonx »

dragontamer wrote: post #37
kraghen wrote: post #14

Earlier quoted context omitted.

It gets even worse: (1 + 2 ^ 53) - 2 ^ 53 evaluates to 0 while 1 + (2 ^ 53 - 2 ^ 53) evaluates to 1 (the correct result), which means that even the operation of adding three floating point numbers has unbounded relative error in the general case. This is a great source of frustration in computational geometry, where this turns from a quantitative issue to a qualitative one, since if we get the sign of an expression w…

> which means that even the operation of adding three floating point numbers has unbounded relative error in the general case. Its not quite "in the general case". In "any case with subtraction", there is unbounded relative error. And remember that addition with negative numbers can become subtraction. Error-management is very important. But if you can GUARANTEE that your operations have the same sign, and that you'r…

It is true that cancellation errors are often the source of wrong results.

However, computations involving IEEE floating point arithmetic can go horribly wrong even if there is no addition and no subtraction whatsoever, no divisions, and only a very small number of rounding errors. Here is one such example:

http://people.eecs.berkeley.edu/~wkahan/WrongR.pdf

Even professional users working in this domain will have a hard time to locate the cause of such problems involving IEEE floating point arithmetic. The format is extremely error-prone to use for casual users and experts alike.

Re: Demystifying Floating Point Precision

#39

Post by dragontamer »

zmonx wrote: post #38
dragontamer wrote: post #37

Earlier quoted context omitted.

> which means that even the operation of adding three floating point numbers has unbounded relative error in the general case. Its not quite "in the general case". In "any case with subtraction", there is unbounded relative error. And remember that addition with negative numbers can become subtraction. Error-management is very important. But if you can GUARANTEE that your operations have the same sign, and that you'r…

It is true that cancellation errors are often the source of wrong results. However, computations involving IEEE floating point arithmetic can go horribly wrong even if there is no addition and no subtraction whatsoever, no divisions, and only a very small number of rounding errors. Here is one such example: http://people.eecs.berkeley.edu/~wkahan/WrongR.pdf Even professional users working in this domain will have a h…

Ehhh... I think that paper kinda "tricks" you with point #3.

> Only 257 algebraic operations, so no hordes of rounding errors

Rounding errors in Floating-point math grows exponentially in the common case. So 257 operations is more than enough to wipe out 53-bits of precision. Its not "hordes" of rounding errors that cause issues. Its the exponential nature of rounding errors, exponentially increasing every step of the way.

Re: Demystifying Floating Point Precision

#40

Post by lucio »

yodacola wrote: post #24

I try to avoid using floats when doing business calculations. Base 2 doesn't work well with our base 10 currency. Real world example: 1.09 - 1.09375 Double: 0.00374999999999992 Decimal: 0.00375

An easier example: type this in your browser console:

0.1 + 0.2 == 0.3

false!(?)

Post reply on HN