John Gustafson’s crusade to replace floating point with something better • Hacker News Board

Re: John Gustafson’s crusade to replace floating point with something better

#151

Post by piadodjanho » Tue, Jul 09, 2019, 3:17 PM UTC

vanderZwan wrote: post #145

The curious skeptics here might want to check out the videos from CoNGA'19[0]. There are talks about posits being used and tried out in the wild with impressive results. For example, according to Millan Klöwer, 16 bit posits could be accurate enough to replace 64 bit floats in certain climate modelling problems[1]. EDIT: just realized that the example was mentioned in the article, with a link to the slides. Still, th…

The innovative idea from posits are they use Golomb-Rice prefix to encode exponent numbers. The Golomb Rice prefix let you encode exponent closer to zero using less space.

For example, the posit16 with nbits=16 and es=1 encodes the exponent like:

01.0 = 0 01.1 = 1 001.0 = 2 001.1 = 3 0001.0 = 4

The format has a normal exponent field with es bits that encodes exponent in binary. When it overflows, it encodes the carry as unnary format. In the unnary format the run-lenght of the same number encodes a number. For example: 0001 would be 3, 001 would be 2, etc. Of course, the posit format is a bit bore complicate than that (it supports negative exponent, for example). But the ideia is pretty much this.

Because of this encoding, if you are working with numbers that have small magnitude posit will have a LOT of more precision than your floating point format.

But the claim that posit16 can have as much precision as binary64 from ieee-754 is misleading. The posit16 can have up to 16-1-2-1=12 bits of precision. While binnary64 always has 53 bits of precision.

They likely compared the binnary64 with posit16 using the accumulator (aka quire). I'm not sure how the quire would map to real world FPU, it uses a lot os space.

Re: John Gustafson’s crusade to replace floating point with something better

#152

Post by ScottFree » Tue, Jul 09, 2019, 3:29 PM UTC

antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

> Very few developers truly understand floating point representation.

Where would one go to better understand how floating points are represented?

Re: John Gustafson’s crusade to replace floating point with something better

#153

Post by piadodjanho » Tue, Jul 09, 2019, 3:34 PM UTC

Gustafnot wrote: post #106

xvilka wrote: post #4

Here is the official site[1] of the project. There is a request[2] to add it in the Scryer Prolog[3] (ISO Prolog implementation in Rust). And implementations in Rust[4] itself, and Julia[5] language. [1] https://posithub.org/index [2] https://github.com/mthom/scryer-prolog/issues/6 [3] https://github.com/mthom/scryer-prolog [4] https://gitlab.com/burrbull/softposit-rs [5] https://juliacomputing.com/blog/2016/03/29/un…

He will keep failing to replace IEEE floating point as long as he insists on making NEGATIVE infinity the same as POSITIVE infinity. Also, IEEE 754 floating point standard guarantees the results of addition, subtraction, multiplication, division, and square root to be the exact correctly rounded value, ie a deterministic result, contrary to what he says.

He argues that underflow/overflow are usually caused by bugs in the code. Having so many special numbers implies reducing the numeric representation (there are a lot of NaN in IEEE-754) and a hardware overhead to deal with all the special cases. That small hardware overhead can add up when working with thousands of FPU.

The vast majority of application dosen't need fine control of the FPU. There is always will be hardware for the few application need the ieee-754 features.

Re: John Gustafson’s crusade to replace floating point with something better

#154

Post by mark-r » Tue, Jul 09, 2019, 3:42 PM UTC

dmd wrote: post #125

rrss wrote: post #124

Earlier quoted context omitted.

Replacing all the IEEE 754 hardware with posits won't fix this, though. If you don't care about performance, then the actual solution has no dependency on hardware: 1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format. 2. Teach people using other languages about floating point and how they may want to use integers instead. The end. No multi-g…

I'm pretty sure that 1.0 + 2.0 == 3.0 in IEEE 754. :) Now, 0.1 ...

One of the great qualities of IEEE 754 is its ability to represent many integers and operate on them without rounding errors.

Re: John Gustafson’s crusade to replace floating point with something better

#155

Post by carapace » Tue, Jul 09, 2019, 3:48 PM UTC

toolslive wrote: post #122

DannyB2 wrote: post #118

Earlier quoted context omitted.

Considering that back in the 1980's we figured out that you should use some kind of 'integer' or 'bcd' based type for financial calculations; it is astonishing that by almost 2020 people are making these same mistakes over and over again. A 64-bit integer is big enough to express the US National Debt in Argentine Pesos.

> it is astonishing that by almost 2020 people are making these same mistakes over and over again. It's actually logical: the number of developers doubles roughly every 5 years. It means that half of the developers have less than 5 years of experience. If they don't teach you this in school (university), you will have to learn from someone who knows, but chances are the other developers are as clueless as you.

OMG! The Eternal Eternal September.

Re: John Gustafson’s crusade to replace floating point with something better

#156

Post by enriquto » Tue, Jul 09, 2019, 3:48 PM UTC

piadodjanho wrote: post #147

enriquto wrote: post #62

Posits and other floating variants are seriously cool, and Gustafson work is amazing. Sadly, the guy has a very annoying writing style that makes him sound like a crackpot. The advantages of posits would shine much more if they were not mixed with ridiculous language (posits are floating point numbers, thus they cannot replace them) and outlandish claims (IEEE floating point is deterministic, to the apparent contradi…

I am implementing a high performance posit arithmetic unit for uni thesis. The idea of using Golomb-Rice prefix tree to encode the exponent is genius. Having said that. I totally agree his papers make a lot of bogus claim. For instance, the claim that "posits can be a direct replacement of floating point" is not true. There are a few tricks, such the Fast Invertion Square Root that abuses the floating point format to…

I'm eager to read your thesis, then! Please, make sure to include a (short) historical account from a neutral point of view. Most notably, Kahan's rejection of unums/posits mixes mostly reasonable criticism with a few points that are not relevant; and then Gustafson holds on to these irrelevant points and replies with his usual word salad. It is quite a sad state of affairs for what could be an enlightening scientific discussion.

The Golomb-Rice prefix tree for the exponents is certainly cool. It has the air of Gosper's "continued logarithms", which are like the same idea but turned to the max: use always the same mantissa (which is implicit) and represent only the exponent to a high precision.

Re: John Gustafson’s crusade to replace floating point with something better

#157

Post by jhj » Tue, Jul 09, 2019, 3:49 PM UTC

ambrop7 wrote: post #52

> It also does away with rounding errors What? Surely it does not have infinite precision with a finite number of bits, and also doesn't seem to be a rational number representation.

That claim can only be made with respect to using a Kulisch accumulator (what he calls the quire) for accumulation.

This can represent the exact sum of any number of floating point values (or products of floating point values) with the intermediate operations preserved at infinite precision, up to a single rounding at the end.

e.g., for computing an inner product of vectors \sum_i x_i y_i with x_i and y_i in floating point, a fused multiply-add operation (as on today's computers) would perform something like:

r(x_n y_n + r(... + r(x_2 y_2 + r(x_1 y_1 + 0))...))

where what is performed within the rounding function r() is done to infinite precision and a single rounding at the end. This is also only true if you are accumulating into the same value, and not splitting this operation across multiple values, otherwise there would be additional rounding steps performed.

Using a Kulisch accumulator the result would be:

r(\sum_i x_i y_i)

similarly also only preserving precision if everything is done in a Kulisch accumulator.

You can add a Kulisch accumulator to IEEE floating point or any other floating point as well. In fact the second? ever computer with floating point (the Zuse Z3) performed FP addition using a similar accumulator, but sums-of-products probably didn't preserve the multiplied value though.

Re: John Gustafson’s crusade to replace floating point with something better

#158

Post by nestorD » Tue, Jul 09, 2019, 3:56 PM UTC

piadodjanho wrote: post #151

vanderZwan wrote: post #145

The curious skeptics here might want to check out the videos from CoNGA'19[0]. There are talks about posits being used and tried out in the wild with impressive results. For example, according to Millan Klöwer, 16 bit posits could be accurate enough to replace 64 bit floats in certain climate modelling problems[1]. EDIT: just realized that the example was mentioned in the article, with a link to the slides. Still, th…

The innovative idea from posits are they use Golomb-Rice prefix to encode exponent numbers. The Golomb Rice prefix let you encode exponent closer to zero using less space. For example, the posit16 with nbits=16 and es=1 encodes the exponent like: 01.0 = 0 01.1 = 1 001.0 = 2 001.1 = 3 0001.0 = 4 The format has a normal exponent field with es bits that encodes exponent in binary. When it overflows, it encodes the carry…

> They likely compared the binnary64 with posit16 using the accumulator (aka quire).

Which I always find deceptiv because nothing stops us from using a quire with classical floating point arithmetic.

It is in fact a comparaison betwen a summation and a compensated summation : it is more precise because the algorithm is different not because of posits or floats.

They tell you that the quire is even faster than a traditional sum while being more precise but reading the associated reference reveals that it only hold with specific hardware which could also be used to use the quire with floats.

(and, arguably, I would love to know that every programmer is aware of a solid implementation of compensated/exact summation/dot-product and uses it when appropriate)

Re: John Gustafson’s crusade to replace floating point with something better

#159

Post by piadodjanho » Tue, Jul 09, 2019, 4:10 PM UTC

antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

Sorry, but you actually sounds like one those people who dosen't really know how FP work.

> I shudder to think how many e-commerce sites use `float` for financial transactions!

The float IEEE-754 represent up to 9 decimal digits (or 23 binary digits) with precision. The double, represent 17 decimal digits. The error upper bound is (0.00000000000000001)/2 per operation. Likely irrelevant for most e-commerce.

Also, the database stores in currency values using fixed point.

> Intel chips have had BCD support in machine code

BCD is floating point encoding not fixed point. AFAIK, only Intel supports it and very precariously.

> I'm surprised nobody has made a decent fixed point lib that is widely used already.

Nonsense. If you do any scientific computation you have likely have Boost, GMP, MPFR installed in your system. They support arbitrary precision arithmetic with integer (aka fixed point), rational and floating point.

Re: John Gustafson’s crusade to replace floating point with something better

#160

Post by piadodjanho » Tue, Jul 09, 2019, 4:14 PM UTC

dmd wrote: post #125

rrss wrote: post #124

Earlier quoted context omitted.

Replacing all the IEEE 754 hardware with posits won't fix this, though. If you don't care about performance, then the actual solution has no dependency on hardware: 1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format. 2. Teach people using other languages about floating point and how they may want to use integers instead. The end. No multi-g…

I'm pretty sure that 1.0 + 2.0 == 3.0 in IEEE 754. :) Now, 0.1 ...

They are not the same thing, but they are close enough most of the time.

The issue with floating point arrises when comparing very close numbers.