Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Examples of floating point problems

jvns.ca

Post reply on HN
Print view View original

61–70 of 179 posts

Re: Examples of floating point problems

#61

Post by zokier »

jordigh wrote: post #57
zokier wrote: post #44

Earlier quoted context omitted.

> 3. You might not like that floats are in binary, which makes decimal arithmetic look weird. But doing decimal arithmetic does not get rid of numerical error, see point 1 (and binary arithmetic thinks your decimal arithmetic looks weird too). One thing that I suspect trips people a lot is decimal string/literal (binary) float conversions instead of the floating point math itself. This includes the classic 0.1+0.2 th…

The only implementation of IEEE754 decimals I've ever seen is in Python's Decimal package. Is there an easily-available implementation anywhere else?

I don't think Pythons Decimal is ieee754, instead its some sort of arbitrary precision thingy.

GCC has builtin support for decimal floats: https://gcc.gnu.org/onlinedocs/gcc/Decimal-Float.html

There are also library implementations floating around, some of them are mentioned in this thread: https://discourse.llvm.org/t/rfc-decimal-floating-point-supp...

decnumber has also rust wrappers if you are so inclined

Re: Examples of floating point problems

#62

Post by thxg »

zokier wrote: post #46
guyomes wrote: post #43

Example 4 mentions that the result might be different with the same code. Here is an example that is particularly counter-intuitive. Some CPU have the instruction FMA(a,b,c) = ab + c and it is guaranteed to be rounded to the nearest float. You might think that using FMA will lead to more accurate results, which is true most of the time. However, assume that you want to compute a dot product between 2 orthogonal vecto…

Note that with gcc/clang you can control the auto-use of fma with compile flags (-ffp-contract=off). It is pretty crazy imho that gcc defaults to using fma

> It is pretty crazy imho that gcc defaults to using fma

Yes! Different people can make different performance-vs-correctness trade-offs, but I also think reproducible-by-default would be better.

Fortunately, specifying a proper standard (e.g. -std=c99 or -std=c++11) implies -ffp-contract=off. I guess specifying such a standard is probably a good idea independently when we care about reproducibility.

Edit: Thinking about it, it the days of 80-bit x87 FPUs, strictly following the standard (specifically, always rounding to 64 bits after every operation) may have been prohibitively expensive. This may explain gcc's GNU mode defaulting to -ffast-math.

Re: Examples of floating point problems

#63

Post by kilotaras »

HarryHirsch wrote: post #59
kilotaras wrote: post #9

Story time. Back in university I was taking part in programming competition. I don't remember the exact details of a problem, but it was expected to be solved as a dynamic problem with dp[n][n] as an answer, n < 1000. But, wrangling some numbers around one could show that dp[n][n] = dp[n-1][n-1] + 1/n, and the answer was just the sum of first N elements of harmonic series. Unluckily for us the intended solution had w…

They didn't take into account that floats come with an estimated uncertainty, and that values that are the same within the limits of experimental error are identical? That's a really badly set problem!

I think it that particular case they just didn't do error analysis.

The task was to output answer with `10^-6` precision, which they solution didn't achieve. Funnily enough the number of other teams went the "correct" route and passed (as they were doing additions in same order as original solution).

Re: Examples of floating point problems

#64

Post by thdc »

saagarjha wrote: post #54
thdc wrote: post #41

Earlier quoted context omitted.

I believe this is technically inaccurate; while Javascript groups most of the number values under, well, "number", modern underlying implementations may resort to perform integer operations when they recognize it is possible. There are also a couple hacks you can do with bit operations to "work" with integers, although I don't remember them off the top of my head - typically used for truncating and whatnot and was ma…

The way runtimes optimize arithmetic is an implementation detail and must conform to IEEE-754.

Fair point, I have been taking smis for granted

Re: Examples of floating point problems

#65

Post by WalterBright »

> NaN/infinity values can propagate and cause chaos

NaN is the most misunderstood feature of IEEE floating point. Most people react to a NaN like they'd react to the dentist telling them they need a root canal. But NaN is actually a very valuable and useful tool!

NaN is just a value that represents an invalid floating point value. The result of any operation on a NaN is a NaN. This means that NaNs propagate from the source of the original NaN to the final printed result.

"This sounds terrible" you might think.

But let's study it a bit. Suppose you are searching an array for a value, and the value is not in the array. What do you return for an index into the array? People often use -1 as the "not found" value. But then what happens when the -1 value is not noticed? It winds up corrupting further attempts to use it. The problem is that integers do not have a NaN value to use for this.

What's the result of sqrt(-1.0)? It's not a number, so it's a NaN. If a NaN appears in your results, you know you've got mistake in your algorithm or initial values. Yes, I know, it can be clumsy to trace it back to its source, but I submit it is better than having a bad result go unrecognized.

NaN has value beyond that. Suppose you have an array of sensors. One of those sensors goes bad (like they always do). What value to you use for the bad sensor? NaN. Then, when the data is crunched, if the result is NaN, you know that your result comes from bad data. Compare with setting the bad input to 0.0. You never know how that affects your results.

This is why D (in one of its more controversial choices) sets uninitialized floating point values to NaN rather than the more conventional choice of 0.0.

NaN is your friend!

Re: Examples of floating point problems

#66

Post by jsmith45 »

zokier wrote: post #38
jordigh wrote: post #16

One thing that pains me about this kind of zoo of problems is that people often have the takeaway, "floating point is full of unknowable, random errors, never use floating point, you will never understand it." Floating point is amazingly useful! There's a reason why it's implemented in hardware in all modern computers and why every programming language has a built-in type for floats. You should use it! And you should…

> One thing that pains me about this kind of zoo of problems is that people often have the takeaway, "floating point is full of unknowable, random errors, never use floating point, you will never understand it." > Floating point is amazingly useful! Another thing about floats is they are for most parts actually very predictable. In particular all basic operations should produce bit-exact results to last ulp. Also bec…

>In particular all basic operations should produce bit-exact results to last ulp.

As long as you are not using a compiler that utilizes x87's extended precision flaots for intermediate calculations, and silently rounding whenever it transfers to memory (That used to be a common issue), and as long as you are not doing dumb stuff with compiler math flags.

Also if you have any code anwhere in your program that relies on correct subnormal handling, then you need to be absolutely sure no code is compiled with `-ffast-math`, including in any dynamically loaded code in your entire program, or your math will break: https://simonbyrne.github.io/notes/fastmath/#flushing_subnor...

And of course if you are doing anything complicated with floating point number, there are entire fields of study about creating numerically stable algorithms, and determining the precision of algorithms with floating point numbers.

Re: Examples of floating point problems

#67

Post by jordigh »

zokier wrote: post #61
jordigh wrote: post #57

Earlier quoted context omitted.

The only implementation of IEEE754 decimals I've ever seen is in Python's Decimal package. Is there an easily-available implementation anywhere else?

I don't think Pythons Decimal is ieee754, instead its some sort of arbitrary precision thingy. GCC has builtin support for decimal floats: https://gcc.gnu.org/onlinedocs/gcc/Decimal-Float.html There are also library implementations floating around, some of them are mentioned in this thread: https://discourse.llvm.org/t/rfc-decimal-floating-point-supp... decnumber has also rust wrappers if you are so inclined

Python's decimal absolutely is IEEE 754 (well, based on the older standard, which has now been absorbed into IEEE 754):

https://github.com/python/cpython/blob/main/Lib/_pydecimal.p...

Cool, didn't know that gcc had built-in support. But is it really as incomplete as it says there?

Re: Examples of floating point problems

#68

Post by astrange »

carapace wrote: post #32
jordigh wrote: post #16

One thing that pains me about this kind of zoo of problems is that people often have the takeaway, "floating point is full of unknowable, random errors, never use floating point, you will never understand it." Floating point is amazingly useful! There's a reason why it's implemented in hardware in all modern computers and why every programming language has a built-in type for floats. You should use it! And you should…

Floating point is a goofy hacky kludge. > There's a reason why it's implemented in hardware in all modern computers Yah, legacy. The reason we used it originally is that computers were small and slow. Now that they're big and fast we could do without it, except that there is already so much hardware and software out there that it will never happen.

Turning all your fixed-size numeric types into variable-sized numeric types introduces some really exciting performance and security issues. (At least if you consider DoS security.)

I think fixed-point math is underrated though.

Re: Examples of floating point problems

#69

Post by zokier »

thxg wrote: post #62
zokier wrote: post #46

Earlier quoted context omitted.

Note that with gcc/clang you can control the auto-use of fma with compile flags (-ffp-contract=off). It is pretty crazy imho that gcc defaults to using fma

> It is pretty crazy imho that gcc defaults to using fma Yes! Different people can make different performance-vs-correctness trade-offs, but I also think reproducible-by-default would be better. Fortunately, specifying a proper standard (e.g. -std=c99 or -std=c++11) implies -ffp-contract=off. I guess specifying such a standard is probably a good idea independently when we care about reproducibility. Edit: Thinking ab…

> Edit: Thinking about it, it the days of 80-bit x87 FPUs, strictly following the standard (specifically, always rounding to 64 bits after every operation) may have been prohibitively expensive

afaik you could just set the precision of x87 to 32/64/80 bits and there would not be any extra cost to the operations

Re: Examples of floating point problems

#70

Post by josefx »

dahfizz wrote: post #34

> Javascript only has floating point numbers – it doesn’t have an integer type. Can anyone justify this? Do JS developers prefer not having exact integers, or is this something that everyone just kinda deals with?

You can use doubles to store and calculate exact integer values. You just wont get 2^64 integers, instead you get the range +/-2^53 .
Post reply on HN