Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

John Gustafson’s crusade to replace floating point with something better

nextplatform.com

Post reply on HN
Print view View original

81–90 of 201 posts

Re: John Gustafson’s crusade to replace floating point with something better

#81

Post by w0utert »

fhars wrote: post #17
savant_penguin wrote: post #16

"(even the same computation on the same system can produce different results for floats)" wait...what? Is this real?

Not really. I suspect they are referring to the original x87 FP functions with their odd 80 bit FP stack and operations, where the exact result depends on when intermediate results are written to RAM and when they are kept in the FP stack, which might depend on intransparent things like compiler optimisations.

>> where the exact result depends on when intermediate results are written to RAM and when they are kept in the FP stack, which might depend on intransparent things like compiler optimisations.

One thing I've never really been sure about is how context-switching affects this. I've always assumed this could also be a source of non-deterministic floating-point results: when the OS decides to switch threads, it has to save the CPU & FPU state, and restore it when switching back. Since context switches are (practically speaking) non-deterministic on a pre-emptive OS, this would imply truncating 80-bit to 64-bit floating point values could also happen on non-deterministic points in time.

Does anyone know if this is actually true? Or do the full 80-bits get saved/restored on context switches?

Re: John Gustafson’s crusade to replace floating point with something better

#82

Post by yaantc »

yholio wrote: post #75

Do these techniques make it easier or harder to implement hardware floating point units? Storage and transmission speed have progressed exponentially while execution units have become the bottleneck. A floating point format that is 20% denser or more accurate but that requires a 2x number of gate delays to implement is major step backwards, except maybe in highly specialized applications.

From [1]: " The standard 32 bits posit adder is found to be twice as large as the corresponding floating-point adder. Posit multiplication requires about 7 times more LUTs and a few more DSPs for a latency which is 2x worst than the IEEE-754 32 bit multiplier." It's for an FPGA implementation.

This being said, I object to your premise: transmission and memory storage are getting comparatively more costly vs computation. From "Computer Architecture, A Quantitative Approach", 6th edition in figure 1.13 there are some values for the TSMC 45nm process, so hardly the leading edge and it only gets worse with finer nodes: - 32 bits integer multiplication: 3.2 pJ - 32 bits float multiplication: 3.7 pJ - 32 bits read in a small 8 kB SRAM: 5 pJ - external DRAM 32 bits read: 640 pJ

We have a lot of transistor nowadays, and as long as the computation can be pipelined halving the memory accesses could be a win (TBC).

Also on from the same Inria team as [1], and also covering hardware cost but more than this: "Posits: the good, the bad and the ugly": https://hal.inria.fr/hal-01959581v3/document

[1] "Hardware cost evaluation of the posit number system", https://hal.inria.fr/hal-02131982/document

Re: John Gustafson’s crusade to replace floating point with something better

#83

Post by Tuna-Fish »

DubiousPusher wrote: post #46
dragontamer wrote: post #24

Earlier quoted context omitted.

Yes and no. 1. Yes -- On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time). 2. No -- Modern x86 systems use SSE registers, which are 64-bits. There are a whole s…

> Modern x86 systems use SSE registers Are your sure of this? It's my understanding that SSE registers require the use of a special API and standard floating point operations do not use them. But the last time I worked with them was writing a SIMD vector library 7 years ago.

64-bit x86 CPUs are defined to support SSE2 as a minimum, and all 64-bit operating systems use it by default.

For 32-bit systems, you still theoretically have to check, but I think every halfway modern compiler also uses SSE2 as the default target.

Re: John Gustafson’s crusade to replace floating point with something better

#84

Post by Tuna-Fish »

w0utert wrote: post #81
fhars wrote: post #17

Earlier quoted context omitted.

Not really. I suspect they are referring to the original x87 FP functions with their odd 80 bit FP stack and operations, where the exact result depends on when intermediate results are written to RAM and when they are kept in the FP stack, which might depend on intransparent things like compiler optimisations.

>> where the exact result depends on when intermediate results are written to RAM and when they are kept in the FP stack, which might depend on intransparent things like compiler optimisations. One thing I've never really been sure about is how context-switching affects this. I've always assumed this could also be a source of non-deterministic floating-point results: when the OS decides to switch threads, it has to s…

They are saved.

The FPU state is saved using the F(N)SAVE x86 instruction, which stores 108 bytes to memory, which contains all visible FPU state.

Re: John Gustafson’s crusade to replace floating point with something better

#85

Post by yorwba »

coldtea wrote: post #64
enriquto wrote: post #62

Posits and other floating variants are seriously cool, and Gustafson work is amazing. Sadly, the guy has a very annoying writing style that makes him sound like a crackpot. The advantages of posits would shine much more if they were not mixed with ridiculous language (posits are floating point numbers, thus they cannot replace them) and outlandish claims (IEEE floating point is deterministic, to the apparent contradi…

> IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson What about this then, which somebody comments below: "On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time,…

Those are not problems a new number format can fix though. The issues with x87 precision are due to compilers freely converting between different representations without this being indicated in the program.

Posits also can't achieve associativity in all conditions, you need to use "quire" accumulators with higher precision. However, if the decision of using quires were left to the compiler, you'd end up with the exact same problem of unclear precision as for x87. To avoid that, those accumulators will need to be annotated in the code, and straightforward programs that do not use them will continue to exhibit non-associativity.

Re: John Gustafson’s crusade to replace floating point with something better

#86

Post by namibj »

coldtea wrote: post #64
enriquto wrote: post #62

Posits and other floating variants are seriously cool, and Gustafson work is amazing. Sadly, the guy has a very annoying writing style that makes him sound like a crackpot. The advantages of posits would shine much more if they were not mixed with ridiculous language (posits are floating point numbers, thus they cannot replace them) and outlandish claims (IEEE floating point is deterministic, to the apparent contradi…

> IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson What about this then, which somebody comments below: "On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time,…

> >IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson

> "parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results."

> due to fp associativity?

As far as I understand, that's not quite the case (on Fermi+ Nvidia, as previous generations had no real support for proper IEEE single compute).

There are pipelines in parallel, but almost all non-determinism in the order of execution (resulting in fp associativity) is exposed to the programmer, except for atomic instructions. Those can even depend on clock skew across PCIe or NVLink, if they operate on remote memory. NVLink supports remote execution in a recent version (I suggest the microbenchmarking-derived microarchitecture analysis papers for Volta and Turing, even if they are less hands-on the Nervana's maxas (with the SGEMM walk-through, which I really recommend)), and thus any hope of reliably tricking synchronization behavior via undefined opcodes/scheduler instructions trickery is deliberately out of the window.

On IBM POWER9 CPUs they can apparently delegate execution of atomic operations to the host CPU, which might (I don't know enough about POWER9 cache coherency /NUMA protocols) delegate them over the NUMA fabric to a different rack-mount unit bonded into the same NUMA mesh.

TL;DR: Recent Nvidia GPUs do not suffer implicit non-determinism in IEEE floats unless you use hardware atomics.

Non-determinism is exposed and introduced in explicit concurrent data structures [0].

[0]: If you count a datastructure with a simple spinlock guarding it as a concurrent datastructure.

PS: I'll add references / links later on the laptop.

Re: John Gustafson’s crusade to replace floating point with something better

#87

Post by Aardwolf »

Choosing to have -infinity and various nans can be done independently from choosing to add regime bits, one could as well design this system to include them by adding more special values.

I didn't find a resource specifying how many exponent bits to actually use? (The 'es' value)

I don't see anything different in this system than regular floats that would ensure consistent results across machines/compilers/...? It's floats plus unary encoded regime bits to have var length exponent, so everything that can make floats inconsistent can still happen here: non commutativity, different exponent sizes or precisions of intermediate values, different rounding modes, ...

Re: John Gustafson’s crusade to replace floating point with something better

#89

Post by OskarS »

roel_v wrote: post #65
rxhernandez wrote: post #59

Earlier quoted context omitted.

Wrt #3, I've had huge errors (I think on the order of 10e-3) in non linear curve fitting spectroscopy algorithms because of this. One of the physics research fellows in the group looked at me like I was an idiot for not knowing the order of multiplication mattered (I still have no clue why he would think it's a standard thing to know this).

Lots of researchers cargo cult floating point programming. I had a Fortran program a few months ago where the author did "if var > 0.99 and var < 1.01 then" etc., where "var" was an integer. I tried to search back where and how that was ever done, but no scenario made any sense ("var" was a categorical variable and always had been). So I went back to some of the original authors and there too they looked at me like I…

In all likelyhood, the reason that's there isn't because a programmer was an idiot. Almost certainly, that variable was originally a float, then some time later another programmer came in and refactored it into an int, but missed fixing this condition, since it technically "works" for ints as well.

This is a good argument for stricter typechecking than anything else. In Rust, a condition like that wouldn't compile because of the mixed types. That's probably a good call in my opinion: it alerts the programmer that there's something funky going on, and it forces the programmer to be explicit about how the comparison should actually happen.

I do find myself doing the "cargo cult" thing from time to time. I was working on a codebase where Lua was embedded in C++, and there were all these flags that had to work in both systems. Because Lua only supports 64-bit floats as a numeric type, the natural type for these were doubles, and "by convention" they only ever had the values 0.0 or 1.0. It was a bit silly of me, but everytime i saw the line `if (fSomeFlag == 1.0f) ...` in the code, I would wince. It just feels wrong to do that to floats. I found myself regularly writing `if (fSomeFlag > 0.5f) ...` instead.

Of course, I was wrong about this. These weren't values determined as a result of arithmetic, they were always set to whole numbers. There was never going to be a case where the actual value of the flag would be 0.9999998f or whatever. But still: once you've been burned by the "floating point equality comparison", you live in fear of it forever after.

Re: John Gustafson’s crusade to replace floating point with something better

#90

Post by mcv »

This sounds really good. I find floating points completely unusable for any situation where accuracy is important. It's fine when being vaguely in the right ballpark is good enough, but I don't want to have to deal with 2 + 4.1 = 6.1000000001, or x/1000000 + y/1000000 != (x+y)/1000000.
Post reply on HN