John Gustafson’s crusade to replace floating point with something better • Hacker News Board

Re: John Gustafson’s crusade to replace floating point with something better

#131

Post by Gustafnot » Tue, Jul 09, 2019, 2:23 PM UTC

FabHK wrote: post #126

Gustafnot wrote: post #106

Earlier quoted context omitted.

He will keep failing to replace IEEE floating point as long as he insists on making NEGATIVE infinity the same as POSITIVE infinity. Also, IEEE 754 floating point standard guarantees the results of addition, subtraction, multiplication, division, and square root to be the exact correctly rounded value, ie a deterministic result, contrary to what he says.

Not so sure. I mean, yes, what you say is right, but there are problems nevertheless, see eg Wikipedia: > Reproducibility > The IEEE 754-1985 allowed many variations in implementations (such as the encoding of some values and the detection of certain exceptions). IEEE 754-2008 has strengthened up many of these, but a few variations still remain (especially for binary formats). The reproducibility clause recommends th…

Reproducibility is an orthogonal issue to Posits vs IEEE754.

Most developers prefer speed over reproducibility, and are encouraged to use denormal-to-zeros, fast-math optimizations, fused MAC, approximated square root function and whatever else is available to achieve results.

The IEEE754 standard provides a guarantee for deterministic results, and many multi-precision and interval arithmetic libraries depend on this guarantee to be true to function properly.

IEEE754 defines unique -infinity and +infinity values, and any "new and improved" standard that breaks this axiom is just incompatible with all existing floating-point libraries written in the last +30 years.

"These claims pander to Ignorance and Wishful Thinking." Kahan (main author of IEEE754) on Posits claims.

Re: John Gustafson’s crusade to replace floating point with something better

#132

Post by FabHK » Tue, Jul 09, 2019, 2:25 PM UTC

epr wrote: post #121

dragontamer wrote: post #24

Earlier quoted context omitted.

Yes and no. 1. Yes -- On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time). 2. No -- Modern x86 systems use SSE registers, which are 64-bits. There are a whole s…

Common Lisp over here like: * (+ (+ 1 1) (expt 2 256)) 115792089237316195423570985008687907853269984665640564039457584007913129639938 * (+ 1 (+ 1 (expt 2 256))) 115792089237316195423570985008687907853269984665640564039457584007913129639938

Many languages support BigFloats and BigInts, but this discussion is about IEEE 754 floats vs Gustavson's proposals.

Re: John Gustafson’s crusade to replace floating point with something better

#133

Post by hinkley » Tue, Jul 09, 2019, 2:25 PM UTC

ambrop7 wrote: post #52

> It also does away with rounding errors What? Surely it does not have infinite precision with a finite number of bits, and also doesn't seem to be a rational number representation.

One form of rounding error that comes up in floating point conversations has to do with data aggregation. If the average value of a series is 3.5 but after rounding the average drops to 3.4, then your rounding has lost some fidelity, and this upsets some people. This counts as a rounding error to them.

Re: John Gustafson’s crusade to replace floating point with something better

#134

Post by PaulHoule » Tue, Jul 09, 2019, 2:28 PM UTC

antonyme wrote: post #111

Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions! So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitf…

The use of binary in the numerator is not the problem with using floats for financial math, it is the use of powers of 2 in the denominator.

The numerator is just an integer and integers are just integers and the base doesn't matter. But if the exponent is base 2, then you can have 1/2, 1/4, 1/8 on the base but not 1/5 or 1/10.

Re: John Gustafson’s crusade to replace floating point with something better

#135

Post by hinkley » Tue, Jul 09, 2019, 2:33 PM UTC

FabHK wrote: post #129

jacobolus wrote: post #9

Earlier quoted context omitted.

The ideas are (1) gradual instead of hard overflow [IEEE floats do gradual underflow via denormals], (2) crowding representable numbers more precisely around 1 while letting very large or very small numbers get less and less precise, vs. floating point which (except for denormals) is scale free within its range, (3) making a format which can be extended by just adding bits without new definitions, so that you can get…

> floating point which (except for denormals) is scale free within its range Presumably that's what you mean by scale free, but let's spell it out: It's denser around 0, then has uniform density between, say, 1/8 and 1/4, half that density between 1/4 and 1/2, half that between 1/2 and 1, half that between 1 and 2, etc. etc. If x = 1.0e22, you can add a million and it's still the same.

I recall in some of these earlier conversations, people pointing out that there are game engines where they invert some of their calculations to prevent artifacts caused by loss of significant figures with certain numbers.

Re: John Gustafson’s crusade to replace floating point with something better

#136

Post by dilawar » Tue, Jul 09, 2019, 2:35 PM UTC

Is there any FPGA implementation of his ideas?

One can then solve some ODEs to check how these implementation are performing.

Ofcourse the speed will be far slower but one can get number of instructions.

Re: John Gustafson’s crusade to replace floating point with something better

#137

Post by vanderZwan » Tue, Jul 09, 2019, 2:35 PM UTC

gwbas1c wrote: post #109

> Better yet, he claims the new format is a “drop-in replacement” for standard floats, with no changes needed to an application’s source code. Claims like that are best taken with a grain of salt: > It also does away with rounding errors, overflow and underflow exceptions, subnormal (denormalized) numbers, and the plethora of not-a-number (NaN) values. Additionally, posits avoids the weirdness of 0 and -0 as two dist…

> But, assuming my application is doing non-trivial math, it's probably aware of the above edge cases. Thus, dropping in posits might have lots of weird side effects where I had to work around weird side effects of floats.

Yeah, but posits appear to have fewer weird edge-cases. Plus the people trying out posits are constantly trying to find methods to work around the limitations too.

During this years conference on posit maths Florent de Dinechin had a really nice talk bringing up all current issues with posits and ways that floating point maths had found workarounds over the year, as a kind of challenge to make posits catch up[0][1]. The community took it really well, as far as I can see, and Gustafson in particular seemed delighted because he genuinely wants everyone to start using better numerical methods.

[0] https://www.youtube.com/channel/UCOstJ2IVC4Y8mbgN0IsowKw

[1] https://www.youtube.com/watch?v=tcX2nRCdZvs

Re: John Gustafson’s crusade to replace floating point with something better

#138

Post by FabHK » Tue, Jul 09, 2019, 2:36 PM UTC

theamk wrote: post #30

shaklee3 wrote: post #20

Earlier quoted context omitted.

I've never heard of that happening on a GPU. Can you give an example?

We have this in our GPU computations, this is very annoying. The errors are usually too small to matter, but they make comparing binary results / reproducible builds impossible. Here is an example I just made: $ python3 >>> s = [10**-x for x in range(16)] >>> s [1, 0.1, 0.01, 0.001, 0.0001, 1e-05, 1e-06, 1e-07, 1e-08, 1e-09, 1e-10, 1e-11, 1e-12, 1e-13, 1e-14, 1e-15] >>> sum(s) 1.1111111111111112 >>> sum(reversed(s))…

Summation of an array is surprisingly non-trivial. Below a discussion about it with many interesting links (including Kahan summation and pairwise summation) if you want to go down some unexpected rabbit holes.

Somewhere I've seen a little routine (by Stefan Karpinski of Julia fame) that starts with a fixed pre-determined array of numbers, and you give it a desired target number, and it permutes the numbers of the array such that when you sum them up (naively, from left to right) the result is your desired target number [2].

https://discourse.julialang.org/t/accurate-summation-algorit...

EDIT: [2] https://discourse.julialang.org/t/array-ordering-and-naive-s...

Re: John Gustafson’s crusade to replace floating point with something better

#139

Post by rrss » Tue, Jul 09, 2019, 2:36 PM UTC

dmd wrote: post #125

rrss wrote: post #124

Earlier quoted context omitted.

Replacing all the IEEE 754 hardware with posits won't fix this, though. If you don't care about performance, then the actual solution has no dependency on hardware: 1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format. 2. Teach people using other languages about floating point and how they may want to use integers instead. The end. No multi-g…

I'm pretty sure that 1.0 + 2.0 == 3.0 in IEEE 754. :) Now, 0.1 ...

Yep, sorry, meant to put 0.1 ...

Thanks!

Re: John Gustafson’s crusade to replace floating point with something better

#140

Post by vanderZwan » Tue, Jul 09, 2019, 2:37 PM UTC

Lerc wrote: post #98

yorwba wrote: post #85

Earlier quoted context omitted.

Those are not problems a new number format can fix though. The issues with x87 precision are due to compilers freely converting between different representations without this being indicated in the program. Posits also can't achieve associativity in all conditions, you need to use "quire" accumulators with higher precision. However, if the decision of using quires were left to the compiler, you'd end up with the exac…

The observation that all of the +-*/ operations in a basic block can be reduced to a single round with the quire would get you a reasonable way with the compiler alone. Overall I'm not sold on the quire thing. It seems like a great big accumulator is a solution that is fairly independent from the floating point format.

> It seems like a great big accumulator is a solution that is fairly independent from the floating point format.

It could still be a good idea to mandate a standard version in the spec for consistent behavior.