Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Floating Point Visually Explained (2017)

fabiensanglard.net

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Floating Point Visually Explained (2017)

#41

Post by bkanber »

bogomipz wrote: post #4
commandlinefan wrote: post #2

Wow, that's a much easier way to convert from decimal to floating point than I had ever seen. He doesn't mention why biased notation is used (i.e. why the exponent is stored as 127+E): it's used so that if you sort positive numbers as if they were integers, they'll still end up in the right order.

>"He doesn't mention why biased notation is used (i.e. why the exponent is stored as 127+E): it's used so that if you sort positive numbers as if they were integers, they'll still end up in the right order." Could you elaborate on this? Maybe an example? This sounds interesting but I'm failing to grasp it.

Well, really the reason for the bias is so that you can get a negative exponent. If E = 10, then the exponent will be 2^-117. You might still need a negative exponent with a positive sign. The offset is used rather than two's complement maybe because of that sorting thing but also because of the math itself, when doing operations on two FP numbers.

Re: Floating Point Visually Explained (2017)

#42

Post by bkanber »

bchociej wrote: post #19

Is all of that really easier to understand than exponential notation? It's a great tool to visualize floating point precision, but it's lot more circuitious to get to an understanding of what a floating point number actually means IMO

The equation fleshes out the detail that the visualization misses. I do best when I have both.

Re: Floating Point Visually Explained (2017)

#43

Post by commandlinefan »

bogomipz wrote: post #4
commandlinefan wrote: post #2

Wow, that's a much easier way to convert from decimal to floating point than I had ever seen. He doesn't mention why biased notation is used (i.e. why the exponent is stored as 127+E): it's used so that if you sort positive numbers as if they were integers, they'll still end up in the right order.

>"He doesn't mention why biased notation is used (i.e. why the exponent is stored as 127+E): it's used so that if you sort positive numbers as if they were integers, they'll still end up in the right order." Could you elaborate on this? Maybe an example? This sounds interesting but I'm failing to grasp it.

Sure, I had to wrestle with this a bit myself: to make it easier, imagine a 16-bit floating point format (P&H call this the "Nvidia format", but I can't find that documented anywhere but there): 1-bit sign, 5-bit (biased) exponent, 10-bit mantissa. One thing that TFA leaves out about floating point mantissas is that there's an implicit leading 1, so a 10-bit mantissa of 1111100000 would be interpreted as (binary) 1.1111100000 or 1 + 2^-1 + 2^-2 + 2^-3 + 2^-4 + 2^-5 = 1.96875. So, take the mantissa, convert it to a fraction, add 1, and then raise it to the power of the (biased) exponent.

So now, take the 16-bit pattern (0000 0011 1110 0000); you get a 0 sign bit, an exponent of 0, and a mantissa of 1.96875. So, with a bias of -15, that's 1.96875 * 2 ^ 0-15 = 0.00006008148193359375.

As you creep up to the "next" exponent, you see that the boundaries are respected. The last 2^-15 number is 0000 0011 1111 1111 (0x3ff) and the first 2^-14 number is 0000 0100 0000 0000 (0x400); now the exponent has changed from 0 to 1, but the floating point converts to 1.999023 * 2 ^ -15 = 0.0000610053539276123046875 and 1.000000 * 2 ^ -14 = 0.00006103515625: a bit-by-bit comparison has 0x3ff Now imagine that IEEE 754 stored exponents in two's-complement format instead; the exponent 01111 would be interpreted as +31, but the "next" exponent, bit-wise, would be 10000 = -32. This means that you'd end up with 0011111111111111 = 1.999023 * 2 ^ 31 = 4292869204.475904, but the next binary number, 0100000000000000 would be 1.0 * 2 -32 = 0.00000000023283.

Re: Floating Point Visually Explained (2017)

#44

Post by anaphor »

GuB-42 wrote: post #36
augustt wrote: post #13

Didn't everyone learn scientific notation in high school? It's pretty much exactly that, and you could put the coefficient/exponent into whatever bit pattern you'd like.

It is scientific notation with a twist. The twist is the implicit 1. in the mantissa. There is no easy way to make it work in decimal. So you need to learn how fractional parts work in binary, then move up to scientific notation in binary. Then you probably noticed that all numbers start with 1, so you can knock it off to save space. Oh, and add a special case for zero, because it is the only number that doesn't star…

It's an implicit 1 or 0 (because it's binary floating point). The implicit 0 is for subnormal numbers. That's the part people usually don't explain when they're first introducing it. It's literally {1,0}.xxxxxx where x is also a 1 or 0. I.e. a binary floating point number in scientific notation. I've seen a lot of explanations that kind of gloss over that part of it (not saying you don't understand it, just that even the article doesn't make that clear enough IMO).

Re: Floating Point Visually Explained (2017)

#45

Post by bkanber »

saagarjha wrote: post #6

Related, https://float.exposed/ is a great resource: both when trying to see how floating point is laid out, as well as when having to convert between the bit representation and the number for "actual work" ;)

That is great. Also helps remind you that for very large numbers, the precision is in the hundreds-of-thousands.

Re: Floating Point Visually Explained (2017)

#46

Post by bigdict »

jonplackett wrote: post #39

Whenever I read something about how computers _really_ work (as in not just a nice easy to comprehend programming language) I realise just how much smarter some people in the world are than me.

With every thing like that that you read, you get closer to them!

Re: Floating Point Visually Explained (2017)

#47

Post by zoomablemind »

It'd be nice to also mention about ulp [0], the unit of least precision. The floating point is an approximation as concept, and ulp is one of the properties of an implementation of floating point representation in binary form.

0: https://stackoverflow.com/questions/43965347/ulp-unit-of-lea...

Re: Floating Point Visually Explained (2017)

#48

Post by piadodjanho »

anaphor wrote: post #44
GuB-42 wrote: post #36

Earlier quoted context omitted.

It is scientific notation with a twist. The twist is the implicit 1. in the mantissa. There is no easy way to make it work in decimal. So you need to learn how fractional parts work in binary, then move up to scientific notation in binary. Then you probably noticed that all numbers start with 1, so you can knock it off to save space. Oh, and add a special case for zero, because it is the only number that doesn't star…

It's an implicit 1 or 0 (because it's binary floating point). The implicit 0 is for subnormal numbers. That's the part people usually don't explain when they're first introducing it. It's literally {1,0}.xxxxxx where x is also a 1 or 0. I.e. a binary floating point number in scientific notation. I've seen a lot of explanations that kind of gloss over that part of it (not saying you don't understand it, just that even…

In practice, subnormal are very rarely used. Most compiler disable subnormals when compiling with anything other than -O0. It takes over a hundred cycle to complete an operation.

Demo: #include 

    int
    main ()
    {
    
        volatile float v;
        float acc = 0;
        float den = 1.40129846432e-45;
    
        for (size_t i; i
With -01: $ gcc float.c -o float -O1 && time ./float ./float 8.93s user 0.00s system 99% cpu 8.933 total

With -O0: $ gcc float.c -o float -O1 && time ./float ./float 20.60s user 0.00s system 99% cpu 20.610 total

Re: Floating Point Visually Explained (2017)

#49

Post by cokernel_hacker »

thechao wrote: post #40
piadodjanho wrote: post #34

Earlier quoted context omitted.

Ackchyually... The IEEE-754 has a lot of redundant representation. Not where you would expect though. Caveat: Those features are invaluable for some niche applications, but not for the average joe. To start. Every IEEE-754 float has two zero representation: one for positive zero and another negative negative zero (sic). The special numbers are another source of redundancy. The the double format, have about 9,007,199,…

I'm mixed on Gustafson's posit stuff. For me, the only thing I'd change for fp would be: 1. -0 now encodes NAN. 2. +inf/-inf are all Fs with sign: 0x7FFFFFFF, 0xFFFFFFFF. 3. 0 is the only denorm. Which does four good things: 1. Gets rid of the utter insanity which is -0. 2. Gets rid of all the redundant NANs. 3. Makes INF "look like" INF. 4. Gets rid of "hard" mixed denorm/norm math. And one seriously bad thing: 1. L…

Interesting. One issue is treatment of 1 / -inf. This would be -0 in traditional IEEE 754 but would now be +0 IIUC.

This would imply that 1 / (1 / -inf) would now be +inf instead of -inf.

Re: Floating Point Visually Explained (2017)

#50

Post by randyrand »

romwell wrote: post #35
randyrand wrote: post #28

Earlier quoted context omitted.

It's not exactly scientific notation because a leading bit is assumed for normalized numbers. The window and offset explanation naturally accounts for the leading bit, whereas the scientific notation explanation needs further explanation to explain how the leading bit works. In short, 10^2 * .001 is not allowed in floating point. You can't have a mantissa that starts with 0. The leading bit means all mantissas must b…

I think this is worth saying about the leading bit (why 0.001 is not a valid mantissa): In binary, we always know the first non-zero digit of any number - so there's no need to write it down. We know it's 1 because, well, binary. So we don't waste space, and only write down all the other digits, and then use the exponent to put the 'point' into the right place. We save a bit of space doing this.

Thats not the right way of thinking about it. Sure, the first non-zero digit of any binary number is 1, but who is to say that the number has a bit that isn't 0? Couldn't the number be zero?

The mantissa can only be a value from [1,2).

i.e. in scientific notation: exponent * mantissa, the mantissa cannot be less than 1, or >= 2 in floating point.

So given that the mantissa can go from 000000 to 111111, what should the values represnet? Obviously it should represent the values from [1,2). Calling it a leading bit is more confusing than it needs to be. Its better to just call it an assumed minimum value.

Post reply on HN