FP2 spec: 00 -> 0.0 01 -> 1.0 10 -> Inf 11 -> NaN or 00 -> 0.0 01 -> 1.0 10 -> Inf 11 -> -Inf
I guess my first car's four speed box was a bit like a FP2 float. Lever forward/back, right/left -> 3.65, 2.15, 1.42, 1.00 ratios.
4-bit floating point FP4
41–50 of 81 posts
Re: 4-bit floating point FP4
#42I too want fewer bits of mantissa in my floating point!
But what I wish is that there had been fp64 encoding with a field for number of significant digits.
strtod() would encode this, fresh out of an instrument reading (serial). It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers.
Every day I get a query like "why does the datum have so many decimal digits? You can't possibly be saying that the instrument is that precise!"
Well, it's because of sprintf(buf, "%.16g", x) as the default to CYA.
Also sad is the complaint about "0.56000 ... 01" because someone did sprintf("%.16f").
I can't fix this in one class -- data travels between too many languages and communication buffers.
In short, I wish I had an fp64 double where the last 4 bits were ALWAYS left alone by the CPU.
Re: 4-bit floating point FP4
#43When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers. As a bonus, any operation can be replaced with a lookup into a nxn table.
You want to make multiplication cheap, it's not just about compression
Multiplication at this resolution is already implemented via lookup tables.
Re: 4-bit floating point FP4
#44I too want fewer bits of mantissa in my floating point! But what I wish is that there had been fp64 encoding with a field for number of significant digits. strtod() would encode this, fresh out of an instrument reading (serial). It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers. Every day I get a query like "why does the datum have so many decimal digits? Yo…
I've seen more packages that do interval arithmetic than those which keep track of significant digits. For example: https://github.com/JuliaIntervals/IntervalArithmetic.jl
Re: 4-bit floating point FP4
#45When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers. As a bonus, any operation can be replaced with a lookup into a nxn table.
That's a good idea and it exists: https://www.johndcook.com/blog/2026/04/18/qlora/
It seems quite wastful to have two zeros when you only have 4 bits it total
Re: 4-bit floating point FP4
#46When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers. As a bonus, any operation can be replaced with a lookup into a nxn table.
That's a good idea and it exists: https://www.johndcook.com/blog/2026/04/18/qlora/ It seems quite wastful to have two zeros when you only have 4 bits it total
OTOH, it seems quite plausible that the most important numbers to represent are:
+0
-0
+1
-1
+inf
-infRe: 4-bit floating point FP4
#47Earlier quoted context omitted.
You want to make multiplication cheap, it's not just about compression
Multiplication at this resolution is already implemented via lookup tables.
For FP4, yes... sometimes... it depends. But newer Nvidia architecture eg Blackwell w/ NVFP4 does not, they perform micro block scaling in the core. For older architectures, low quants like FP4 are also often not done native, and instead inflated back to BF16, eg with BnB.
Re: 4-bit floating point FP4
#48Earlier quoted context omitted.
That's a good idea and it exists: https://www.johndcook.com/blog/2026/04/18/qlora/ It seems quite wastful to have two zeros when you only have 4 bits it total
OTOH, it seems quite plausible that the most important numbers to represent are: +0 -0 +1 -1 +inf -inf
Why waste a slot on -0?
Re: 4-bit floating point FP4
#49Earlier quoted context omitted.
I especially like your HQQ precision
I think it is only a matter of time before HQQ / 1FP takes over. It's the logical conclusion. I hope to be using my 96-blade razor by then too
Re: 4-bit floating point FP4
#50Earlier quoted context omitted.
That's a good idea and it exists: https://www.johndcook.com/blog/2026/04/18/qlora/ It seems quite wastful to have two zeros when you only have 4 bits it total
OTOH, it seems quite plausible that the most important numbers to represent are: +0 -0 +1 -1 +inf -inf
In standard FP32, the infs are represented as a sign bit, all exponent bits=1, and all mantissa bits=0. The NaNs are represented as a sign bit, all exponent bits=1, and the mantissa is non-zero. If you used that interpretation with FP4, you'd get the table below, which restricts the representable range to +/- 3, and it feels less useful to me. If you're using FP4 you probably are space optimized and don't want to waste a quarter of your possible combinations on things that aren't actually numbers, and you'd likely focus your efforts on writing code that didn't need to represent inf and NaN.
Bits s exp m Value
-------------------
0000 0 00 0 +0
0001 0 00 1 +0.5
0010 0 01 0 +1
0011 0 01 1 +1.5
0100 0 10 0 +2
0101 0 10 1 +3
0110 0 11 0 +inf
0111 0 11 1 NaN
1000 1 00 0 -0
1001 1 00 1 -0.5
1010 1 01 0 -1
1011 1 01 1 -1.5
1100 1 10 0 -2
1101 1 10 1 -3
1110 1 11 0 -inf
1111 1 11 1 NaN