Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Examples of floating point problems

jvns.ca

Post reply on HN
Print view View original

41–50 of 179 posts

Re: Examples of floating point problems

#41

Post by thdc »

dahfizz wrote: post #34

> Javascript only has floating point numbers – it doesn’t have an integer type. Can anyone justify this? Do JS developers prefer not having exact integers, or is this something that everyone just kinda deals with?

I believe this is technically inaccurate; while Javascript groups most of the number values under, well, "number", modern underlying implementations may resort to perform integer operations when they recognize it is possible. There are also a couple hacks you can do with bit operations to "work" with integers, although I don't remember them off the top of my head - typically used for truncating and whatnot and was mainly a performance thing.

Also there are typed arrays and bigints if we can throw those in, too.

Re: Examples of floating point problems

#42

Post by gugagore »

ogogmad wrote: post #13

Related: In numerical analysis, I found the distinction between forwards and backwards numerical error to be an interesting concept. The forwards error initially seems like the only right kind, but is often impossible to keep small in numerical linear algebra. In particular, Singular Value Decomposition cannot be computed with small forwards error. But the SVD can be computed with small backwards error. Also: The JSO…

IIRC, forward error: the error between the given answer and the right answer to the given question.

Backward error: the error between the given question, and the question whose right answer is the given answer.

Easier to parse like this: a small forward error means that you give an answer close to the right one.

A small backward error means that the answer you give is the right answer for a nearby question.

Re: Examples of floating point problems

#43

Post by guyomes »

Example 4 mentions that the result might be different with the same code. Here is an example that is particularly counter-intuitive.

Some CPU have the instruction FMA(a,b,c) = ab + c and it is guaranteed to be rounded to the nearest float. You might think that using FMA will lead to more accurate results, which is true most of the time.

However, assume that you want to compute a dot product between 2 orthogonal vectors, say (u,v) and (w,u) where w = -v. You will write:

p = uv + wu

Without FMA, that amounts to two products and an addition between two opposite numbers. This results in p = 0, which is the expected result.

With FMA, the compiler might optimize this code to:

p = FMA(u, v, wu)

That is one FMA and one product. Now the issue is that wu is rounded to the nearest float, say x, which is not exactly -vu. So the result will be the nearest float to uv + x, which is not zero!

So even for a simple formula like this, testing if two vectors are orthogonal would not necessary work by testing if the result is exactly zero. One recommended workaround in this case is to test if the dot product has an absolute value smaller than a small threshold.

Re: Examples of floating point problems

#44

Post by zokier »

jordigh wrote: post #16

One thing that pains me about this kind of zoo of problems is that people often have the takeaway, "floating point is full of unknowable, random errors, never use floating point, you will never understand it." Floating point is amazingly useful! There's a reason why it's implemented in hardware in all modern computers and why every programming language has a built-in type for floats. You should use it! And you should…

> 3. You might not like that floats are in binary, which makes decimal arithmetic look weird. But doing decimal arithmetic does not get rid of numerical error, see point 1 (and binary arithmetic thinks your decimal arithmetic looks weird too).

One thing that I suspect trips people a lot is decimal string/literal (binary) float conversions instead of the floating point math itself. This includes the classic 0.1+0.2 thing, and many of the problems in the article.

I think these days using floating point hex strings/literals more would help a lot. There are also decimal floating point numbers that people largely ignore despite being standard for over 15 years

Re: Examples of floating point problems

#45

Post by [deleted] »

guyomes wrote: post #43

Example 4 mentions that the result might be different with the same code. Here is an example that is particularly counter-intuitive. Some CPU have the instruction FMA(a,b,c) = ab + c and it is guaranteed to be rounded to the nearest float. You might think that using FMA will lead to more accurate results, which is true most of the time. However, assume that you want to compute a dot product between 2 orthogonal vecto…

[deleted]

Re: Examples of floating point problems

#46

Post by zokier »

guyomes wrote: post #43

Example 4 mentions that the result might be different with the same code. Here is an example that is particularly counter-intuitive. Some CPU have the instruction FMA(a,b,c) = ab + c and it is guaranteed to be rounded to the nearest float. You might think that using FMA will lead to more accurate results, which is true most of the time. However, assume that you want to compute a dot product between 2 orthogonal vecto…

Note that with gcc/clang you can control the auto-use of fma with compile flags (-ffp-contract=off). It is pretty crazy imho that gcc defaults to using fma

Re: Examples of floating point problems

#47

Post by dunham »

I had one issue where pdftotext would produce different output on different machines (Linux vs Mac). It broke some of our tests.

I tracked down where it was happening (involving an ==), but it magically stopped when I added print statements or looked at it in the debugger.

It turns out the x86 was running the math at a higher precision and truncating when it moved values out of registers - as soon as it hit memory, things were equal. MacOS was defaulting to -ffloat-store to get consistency (their UI library is float based).

There were too many instances of == in that code base (which IMO is a bad idea with floats), so I just added -ffloat-store to the Linux build and called it a day.

Re: Examples of floating point problems

#48

Post by lanstin »

guyomes wrote: post #43

Example 4 mentions that the result might be different with the same code. Here is an example that is particularly counter-intuitive. Some CPU have the instruction FMA(a,b,c) = ab + c and it is guaranteed to be rounded to the nearest float. You might think that using FMA will lead to more accurate results, which is true most of the time. However, assume that you want to compute a dot product between 2 orthogonal vecto…

In general with reals with any source of error anywhere, this caution about equality is always correct. the odds of two reals being equal is zero.

Re: Examples of floating point problems

#49

Post by alkonaut »

dunham wrote: post #47

I had one issue where pdftotext would produce different output on different machines (Linux vs Mac). It broke some of our tests. I tracked down where it was happening (involving an ==), but it magically stopped when I added print statements or looked at it in the debugger. It turns out the x86 was running the math at a higher precision and truncating when it moved values out of registers - as soon as it hit memory, t…

x86 (x87) FP is notoriously inconsistent because of the 80 bit extended precision that may not be used. In a JITed language line Java/C# it’s even less fun as it can theoretically be inconsistent even for the same compiled program on different machines.

Thankfully the solution to that problem came when x86 (32 bit) mostly disappeared.

Re: Examples of floating point problems

#50

Post by sacrosancty »

svat wrote: post #22

If you have only a couple of minutes to develop a mental model of floating-point numbers (and you have none currently), the most valuable thing IMO would be to spend them staring at a diagram like this one: https://upload.wikimedia.org/wikipedia/commons/b/b6/Floating... (uploaded to Wikipedia by user Joeleoj123 in 2020, made using Microsoft Paint) — it already covers the main things you need to know about floating-po…

If you have even less time, just think of them as representing physical measurements made with practical instruments and the math done with analog equipment.

The common cause of floating point problems is usually treating them as a mathematical ideal. The quirks appear at the extremes when you try to to un-physical things with them. You can't measure exactly 0 V with a voltmeter, or use an instrument for measuring the distance to stars then add a length obtained from a micrometer without entirely losing the latter's contribution.

Post reply on HN