Earlier quoted context omitted.
OTOH, it seems quite plausible that the most important numbers to represent are: +0 -0 +1 -1 +inf -inf
Why waste a slot on -0?
Because it means "infinitesimal negative" which is distinct from "infinitesimal positive".
4-bit floating point FP4
51–60 of 81 posts
Re: 4-bit floating point FP4
#52Earlier quoted context omitted.
Why waste a slot on -0?
Because it means "infinitesimal negative" which is distinct from "infinitesimal positive".
That sounds pretty niche. What's a use case where you have less than 8 bits and that distinction is more important than having an extra finite value? I don't think AI is one.
Re: 4-bit floating point FP4
#53Earlier quoted context omitted.
Because it means "infinitesimal negative" which is distinct from "infinitesimal positive".
That sounds pretty niche. What's a use case where you have less than 8 bits and that distinction is more important than having an extra finite value? I don't think AI is one.
For neural net gradient descent, automatic differentiation etc, the widely used ReLU function has infornation carrying derivatives at +0 and –0 if those are infinitesimals.
Re: 4-bit floating point FP4
#54> In ancient times, floating point numbers were stored in 32 bits. I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.
That is merely medieval times. In ancient times, floats were all 60 bits and there was no single precision. See page 3-15 of this https://caltss.computerhistory.org/archive/6400-cdc.pdf
I see their 60-bit float has the same size exponent (11 bits) as today's doubles. Only the mantissa was smaller, 48 bits instead of 52.
Re: 4-bit floating point FP4
#55> In ancient times, floating point numbers were stored in 32 bits. Then somewhere along the way 64 bits became standard.
I think Cray doubles were 128 bits, and their singles were 64… which makes it seem like smaller floats are just a continuation of the eternal trend.
Re: 4-bit floating point FP4
#56Earlier quoted context omitted.
That sounds pretty niche. What's a use case where you have less than 8 bits and that distinction is more important than having an extra finite value? I don't think AI is one.
For neural net gradient descent, automatic differentiation etc, the widely used ReLU function has infornation carrying derivatives at +0 and –0 if those are infinitesimals.
Barely any information. After surviving RELU that signed zero is probably getting added to another value and then oops the information is gone. It sounds a lot worse than properly spaced values.
Re: 4-bit floating point FP4
#57Earlier quoted context omitted.
For neural net gradient descent, automatic differentiation etc, the widely used ReLU function has infornation carrying derivatives at +0 and –0 if those are infinitesimals.
Barely any information. After surviving RELU that signed zero is probably getting added to another value and then oops the information is gone. It sounds a lot worse than properly spaced values.
sign = most important bit of information
Re: 4-bit floating point FP4
#58Earlier quoted context omitted.
That sounds pretty niche. What's a use case where you have less than 8 bits and that distinction is more important than having an extra finite value? I don't think AI is one.
For neural net gradient descent, automatic differentiation etc, the widely used ReLU function has infornation carrying derivatives at +0 and –0 if those are infinitesimals.
[deleted]
Re: 4-bit floating point FP4
#59I too want fewer bits of mantissa in my floating point! But what I wish is that there had been fp64 encoding with a field for number of significant digits. strtod() would encode this, fresh out of an instrument reading (serial). It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers. Every day I get a query like "why does the datum have so many decimal digits? Yo…
> It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers.
It would be useful if you could then pass it to an "about equal" operator, too.
I don't need to know that the alternator is putting out 13.928528V, and sure as hell I know you're not measuring that accurately. It's precise but wrong.
I want an "about equals" thing so I can say "if Valt == 14 alt_ok=true" kind of thing but tag it to be "about 14" not "exactly 14".
Re: 4-bit floating point FP4
#60> In ancient times, floating point numbers were stored in 32 bits.
This was true only for cheap computers, typically after the mid sixties.
Most of the earliest computers with vacuum tubes used longer floating-point number formats, e.g. 48-bit, 60-bit or even weird sizes like 57-bit.
The 32-bit size has never been acceptable in scientific computing with complex computations where rounding errors accumulate. The early computers with floating-point hardware were oriented to scientific/technical computing, so bigger number sizes were preferred. The computers oriented to business applications usually preferred fixed-point numbers.
The IBM System/360 family has definitively imposed the 32-bit single-precision and 64-bit double-precision sizes, where 32-bit is adequate for input data and output data and it can be sufficient for intermediate values when the input data passes through few computations, while otherwise double-precision must be used.