Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Gustafson’s crusade to replace floating point with something better

nextplatform.com

Post reply on HN
Print view View original

141–150 of 201 posts

Re: John Gustafson’s crusade to replace floating point with something better

#141

Post by piadodjanho »

antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

In the paper "Do Developers Understand IEEE Floating Point" the authors surveys phd student and faculty members from computer science and found out that your observation is true: most people don't know how fp works.

They have the survey online at [1] in case you want to see how much you know about fp behavior.

[1] http://presciencelab.org/float

Re: John Gustafson’s crusade to replace floating point with something better

#142

Post by mcv »

nestorD wrote: post #120
mcv wrote: post #108

Earlier quoted context omitted.

Really? My impression from the article was that this was exactly one of the things posits were supposed to fix, at least for numbers with small exponents. I'm not sure how 10.0 * 2.0 = 16.0. I'm not sure what posit8 means, but it can only be correct if it switches halfway from base 8 representation to base 10, which is a bit weird, but at least the calculation is correct. (Otherwise it would be so incorrect to be unu…

I am serious (and recommend reading the paper I quoted to have a good understanding of the trade-of offered by posits). Overall, posits are a new trade-of that will give you better precision (nothing exact, it is still an approximation) when you manage to keep all your number in a small range. Once you get out of that range precision drops significantly (whereas the precision of classical floating points drops gradua…

If it's just a variation of the same problems behind floats, then I'm not that interested. Well, I guess it depends on how small the range is.

Re: John Gustafson’s crusade to replace floating point with something better

#143

Post by vanderZwan »

Gustafnot wrote: post #131
FabHK wrote: post #126

Earlier quoted context omitted.

Not so sure. I mean, yes, what you say is right, but there are problems nevertheless, see eg Wikipedia: > Reproducibility > The IEEE 754-1985 allowed many variations in implementations (such as the encoding of some values and the detection of certain exceptions). IEEE 754-2008 has strengthened up many of these, but a few variations still remain (especially for binary formats). The reproducibility clause recommends th…

Reproducibility is an orthogonal issue to Posits vs IEEE754. Most developers prefer speed over reproducibility, and are encouraged to use denormal-to-zeros, fast-math optimizations, fused MAC, approximated square root function and whatever else is available to achieve results. The IEEE754 standard provides a guarantee for deterministic results, and many multi-precision and interval arithmetic libraries depend on this…

> Most developers prefer

You're free to voice your own opinion, but I take some issue with people asserting theirs as if they speak for "most developers". Especially if it comes from a new account with a name like "Gustafnot". That doesn't exactly scream "unbiased" to me.

Re: John Gustafson’s crusade to replace floating point with something better

#144

Post by IEEE754 »

rrss wrote: post #124
antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

Replacing all the IEEE 754 hardware with posits won't fix this, though. If you don't care about performance, then the actual solution has no dependency on hardware: 1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format. 2. Teach people using other languages about floating point and how they may want to use integers instead. The end. No multi-g…

> 1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format.

Unlimited precision in any radix point based format does not solve representation error. If you don't understand why:

  - How many decimal places does it take to represent 1/3 (infinite, AKA out of memory)

  - Now how many places does it take to represent 1/3 in base3? (1)
If you are truly only working with rational numbers and only using the four basic arithmetic operations, then only a variable precision fractional representation (i.e a numerator and demonstrator, which is indifferent to underlying base) will be able to store any number without error (if it fits in memory). Of course if you are using transcendental functions or want to use irrational numbers e.g PI then by definition there is no numerical solution to avoid error in any finite system.

Re: John Gustafson’s crusade to replace floating point with something better

#145

Post by vanderZwan »

The curious skeptics here might want to check out the videos from CoNGA'19[0]. There are talks about posits being used and tried out in the wild with impressive results. For example, according to Millan Klöwer, 16 bit posits could be accurate enough to replace 64 bit floats in certain climate modelling problems[1].

EDIT: just realized that the example was mentioned in the article, with a link to the slides. Still, the YT talk may add some context

[0] https://www.youtube.com/channel/UCOstJ2IVC4Y8mbgN0IsowKw/vid...

[1] https://www.youtube.com/watch?v=XazIx0cMVyg

Re: John Gustafson’s crusade to replace floating point with something better

#146

Post by dragontamer »

theamk wrote: post #30
shaklee3 wrote: post #20

Earlier quoted context omitted.

I've never heard of that happening on a GPU. Can you give an example?

We have this in our GPU computations, this is very annoying. The errors are usually too small to matter, but they make comparing binary results / reproducible builds impossible. Here is an example I just made: $ python3 >>> s = [10**-x for x in range(16)] >>> s [1, 0.1, 0.01, 0.001, 0.0001, 1e-05, 1e-06, 1e-07, 1e-08, 1e-09, 1e-10, 1e-11, 1e-12, 1e-13, 1e-14, 1e-15] >>> sum(s) 1.1111111111111112 >>> sum(reversed(s))…

If you don't care about cancellation error and are only concerned with perfect bitwise / reproducible results, then sort your numbers.

Sorting the numbers is necessary to minimize cancellation error. But absolute minimum cancellation error can only be done sequentially: the parallel version may return different results.

    l = list(s)
    l.sort()
    sum(l)
    l = list(reversed(s))
    l.sort()
    sum(l)
Both return 1.111111111111111 now. Sorting is the "synchronization" step you need to get all your parallel tasks back into a defined order.

Re: John Gustafson’s crusade to replace floating point with something better

#147

Post by piadodjanho »

enriquto wrote: post #62

Posits and other floating variants are seriously cool, and Gustafson work is amazing. Sadly, the guy has a very annoying writing style that makes him sound like a crackpot. The advantages of posits would shine much more if they were not mixed with ridiculous language (posits are floating point numbers, thus they cannot replace them) and outlandish claims (IEEE floating point is deterministic, to the apparent contradi…

I am implementing a high performance posit arithmetic unit for uni thesis. The idea of using Golomb-Rice prefix tree to encode the exponent is genius. Having said that. I totally agree his papers make a lot of bogus claim.

For instance, the claim that "posits can be a direct replacement of floating point" is not true. There are a few tricks, such the Fast Invertion Square Root that abuses the floating point format to do fast arithmetic operation. This would break replacing with posit.

Also he claims that early FPGA posit implementation uses less area and is has high performance than a Floating Point implementation. This cannot be true^. The posit unit IS A floating point unit with a encoder/decoder unit. The posit floating point unit will actually have a larger word size when decoded than ieee floating point because it has more precision than ieee format. Also the features such different rounding supported by ieee but not by posit uses a negligible extra area.

^ Unless, ofc, he is comparing apples to oranges.

Re: John Gustafson’s crusade to replace floating point with something better

#148

Post by microcolonel »

antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

Rational numbers are a good way to do many financial calculations (extra points for doing something useful with negative denominators!), since many financial calculations are specified with particular denominators (per cent, per mille, basis points; halves, sixths, twelfths, twenty-sixths, fifty-seconds of a basis point, etc.).

However, as soon as you start doing anything interesting, you have limited precision as a matter of course.

Re: John Gustafson’s crusade to replace floating point with something better

#149

Post by mcv »

antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

If there's one thing I've always really appreciated about Groovy, it's that it used BigDecimal as the default for fractions, because 9 times out of 10, you need accuracy more than you need high performance and large exponents (and if you do need high performance, you wouldn't be using Groovy anyway).

Sadly most languages don't support something like that out of the box.

Re: John Gustafson’s crusade to replace floating point with something better

#150

Post by simonh »

DannyB2 wrote: post #118
antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

Considering that back in the 1980's we figured out that you should use some kind of 'integer' or 'bcd' based type for financial calculations; it is astonishing that by almost 2020 people are making these same mistakes over and over again. A 64-bit integer is big enough to express the US National Debt in Argentine Pesos.

It's not enough to represent one US cent in original Zimbabwean dollars though as of 2009. Although those 'first' dollars hadn't been legal for quite some time, the currency having gone through three rounds of massive devaluation and collapse, totalling something like 1x10^30 by then.
Post reply on HN