Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Floating Point Visually Explained (2017)

fabiensanglard.net

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Floating Point Visually Explained (2017)

#71

Post by piadodjanho »

piadodjanho wrote: post #48
anaphor wrote: post #44

Earlier quoted context omitted.

It's an implicit 1 or 0 (because it's binary floating point). The implicit 0 is for subnormal numbers. That's the part people usually don't explain when they're first introducing it. It's literally {1,0}.xxxxxx where x is also a 1 or 0. I.e. a binary floating point number in scientific notation. I've seen a lot of explanations that kind of gloss over that part of it (not saying you don't understand it, just that even…

In practice, subnormal are very rarely used. Most compiler disable subnormals when compiling with anything other than -O0. It takes over a hundred cycle to complete an operation. Demo: #include int main () { volatile float v; float acc = 0; float den = 1.40129846432e-45; for (size_t i; i With -01: $ gcc float.c -o float -O1 && time ./float ./float 8.93s user 0.00s system 99% cpu 8.933 total With -O0: $ gcc float.c -o…

I looked at the asm generate from my original example and they generate very different codes, gcc applies other optimization when compiled with -O1.

I've been fighting the compiler to generate a minimal working example of the subnormals, but didn't have any success.

Some things take need to be taken in account (from the top of my head):

- Rounding. You don't want to get stuck in the same number. - The FPU have some accumulator register that are larger than the floating point register. - Using more register than the architecture has it not trivial because the register renaming and code reordering. The CPU might optimize in a way that the data never leaves those register.

Trying to make a mwe, I found this code:

    #include 
    
    int
    main ()
    {
        double x = 5e-324;
        double acc = x;
    
        for (size_t i; i
Runs is fraction of seconds with -O0:

    gcc double.c -o double -O0
But takes forever (killed after 5 minutes) with -O1:

    gcc double.c -o double -O1
I'm using gcc (Arch Linux 9.3.0-1) 9.3.0 on i7-8700

I also manage to create a code that sometimes run in 1s, but in others would take 30s. Didn't matter if I recompiled.

Floating point is hard.

Re: Floating Point Visually Explained (2017)

#72

Post by piadodjanho »

thechao wrote: post #67
piadodjanho wrote: post #51

Earlier quoted context omitted.

He also propose the use of an opaque register to accumulate (quire), in contrast to the transparent float register (its a mess, each compiler does what it think is best). When working with numbers that exceed the posit representation you use the quire to accumulate. At the end of the computation you convert again to posit to store in memory, or store the quire in memory. In C, it would look like something like: posit…

I come from GPU-land, and a quire always brings a chuckle from the fp HW folk. They like the rest of Gustafson’s stuff, though.

Yeah. Gustafons is like, memory is cheap, just use a few hundreds of kb to store the quire. To be fair, he is not HW guy.

Re: Floating Point Visually Explained (2017)

#73

Post by piadodjanho »

thechao wrote: post #66
cokernel_hacker wrote: post #49

Earlier quoted context omitted.

Interesting. One issue is treatment of 1 / -inf. This would be -0 in traditional IEEE 754 but would now be +0 IIUC. This would imply that 1 / (1 / -inf) would now be +inf instead of -inf.

0 is unsigned. I would reject 1/inf — it would be NAN. If the user wants to play silly games with derivatives, computer algebra systems are that way: —>.

NaN is NaR (not a real) in posit notation.

Re: Floating Point Visually Explained (2017)

#74

Post by petschge »

splittingTimes wrote: post #52

That's why when I did numerical simulation of electron Dynamics in semiconductors during my phD we never used straight SI units (m, s, kg, etc), but instead expressed all physical natural constants in nm, fs, eV, etc. That way all relevant constants had numerical values between 1 and 10 which stabilized the simulations a lot.

Expressing all quantities as small multiples of suitable reference quantities is quite common. The resulting equations after simplifications are often referred to as dimensionless (because the reference quantity has absorbed the units and only a small dimensionless number remains).

There is two upsides to this: 1.) All quantities are order one, safely away from underflow and infinity. 2.) The equations are generally simpler, which was important when the many limitation in simulation codes was flops, not memory bandwidth.

There are also important downsides: 1.) You have to manipulate the equations before you start coding. Porting features from a code that used another normalization is quite error-prone. 2.) All quantities are order one, which makes it hard to detect if you accidentally use and electric field instead of a magnetic field. All you see is a number of roughly the correct magnitude. 3.) Comparison between codes is more difficult, in particular if the use different normalization, because you have to convert from "code units" to actual SI units.

Re: Floating Point Visually Explained (2017)

#75

Post by saagarjha »

Someone wrote: post #60
saagarjha wrote: post #20

Earlier quoted context omitted.

This has more to do with the mantissa than the biased exponent.

No, it has to do with bothequally. Changing a 0 bit to a 1 in an integer increases its value, and by the rule I gave, should also increase the value of the floating point number with the same bit pattern. It doesn’t matter whether that flipped bit is in the mantissa or the exponent. That requires the use of the biased number in the exponent. In particular, the “all zeroes” bit pattern for the exponent must be the rep…

This is a different claim than the one above, which was about incrementing the least significant bit.

Re: Floating Point Visually Explained (2017)

#76

Post by Veserv »

titzer wrote: post #69
Veserv wrote: post #21

For another visual explanation in words, floating point numbers (ignoring subnormal) are just a linear approximation of 2^x [1] where there is one piece for each integer (x = 4 to x = 5, etc). As an example, draw a straight line between 2^4 (16) and 2^5 (32). The floating point numbers in that range are evenly spaced on that line. Another explanation using the window + offset terminology used in the post is that the…

On the other hand, if you don't have subnormals, then you have the funny property that subtracting two inequal numbers would yield 0. This never happens with subnormals because they allow representing numbers closer to zero, below the smallest representable exponent. The numerical stability of some algorithms crucially relies on subnormals.

I do not understand your statement. I can come up with two possible interpretations, but neither seem true to me. Can you provide an example of what you mean?

I am also interested in your statement that certain algorithms depend on subnormals as defined in the floating point spec. Can you provide an example of such an algorithm? I can intuit how it might be desirable to have a single "too small/epsilon" value, but I do not see offhand how you can leverage the full range of subnormals in any reasonably generic way that is not extremely dependent on the specific problem and scale (i.e. multiply all numbers by 2^30 and increase the maximum exponent by 30, do you get the same output multiplied by 2^30), so I would like to see how it is done.

In terms of my two possible interpretations, the first is that subtracting two unequal floating point numbers yield 0. I am pretty sure this is not the case. It may yield no change, but I am pretty sure it can not yield 0. The other is that subtracting two unequal arbitrary precision numbers represented as floating point numbers yield 0. This is true, but is a known limitation of emulating arbitrary precision arithmetic using limited precision and must always be accounted for. If this is what you meant, then we can just choose numbers too small to be expressed by subnormals to cause the problem to occur again, so all it does is allow handling of a few more cases at the cost of complexity and non-uniformity. If you did not mean either of these two interpretations, can you explain what you meant, preferably with a concrete example?

Re: Floating Point Visually Explained (2017)

#77

Post by mschuetz »

splittingTimes wrote: post #52

That's why when I did numerical simulation of electron Dynamics in semiconductors during my phD we never used straight SI units (m, s, kg, etc), but instead expressed all physical natural constants in nm, fs, eV, etc. That way all relevant constants had numerical values between 1 and 10 which stabilized the simulations a lot.

File formats for georeferenced data also regularly store coordinates as 32 bit ints, with a scaling argument that can be used to convert the integer into a double precision number with a specific unit. E.g. your scene unit is meters and you want to store with a precision of millimeters? Multiply double precision coordinates by 1000, convert the result to an int32, save it to file. When you load the coordinate afterwards, divide it by 1000 and you you get back your double precision coordinate.

Re: Floating Point Visually Explained (2017)

#78

Post by baby »

Interesting, because I think about it the other way: the offset tells you between which power of two you're in:

* [2^-127, 2^-126] * ... * [1, 2] * [2, 2^2] * [2^2, 2^3] * ... * [2^128, 2^129]

and the window tells you where you are in this "window". You have 8 bits to figure out where you are in there, so it's obvious that you have much more precision if you are in the window [1, 2] than in the window [2^128, 2^129]

Re: Floating Point Visually Explained (2017)

#79

Post by totorovirus »

0-_-0 wrote: post #53

Fabien Sargland is the same guy who wrote 2 books on a deep dive into 2 game engines: Wofenstein 3D and Doom which are great reading and I really recommend them to the HN crowd (can be downloaded for free): https://fabiensanglard.net/gebbwolf3d/ https://fabiensanglard.net/gebbdoom/

this looks really interesting! thanks for sharing :)

Re: Floating Point Visually Explained (2017)

#80

Post by enriquto »

piadodjanho wrote: post #37
enriquto wrote: post #15

Earlier quoted context omitted.

The 8-bit floating point is even easier to understand, since you can list them all in a single page, and even visualize the entire addition and multiplication tables.

Also a great format to count how many representation are wasted with redundant representation (ZeroS, NaN, +inf, -inf).

What is redundant about that? It "wastes" a completely negligible part of the representation space, and the consistency gains are enormous.
Post reply on HN