Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

4-bit floating point FP4

johndcook.com

Post reply on HN
Print view View original

71–80 of 81 posts

Re: 4-bit floating point FP4

#71

Post by adrian_b »

childintime wrote: post #34
teo_zero wrote: post #33

When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers. As a bonus, any operation can be replaced with a lookup into a nxn table.

Exactly. And pick them on the e^x curve.

As explained in an article linked at the bottom of TFA, the weights of a LLM have a normal (Gaussian) distribution.

Because of that, the best compromise when the weights are quantized to few levels is to place the points encoded by the numeric format used for the weights using a Gaussian function, instead of placing them uniformly on a logarithmic scale, like the usual floating-point formats attempt.

Re: 4-bit floating point FP4

#72

Post by nomel »

the__alchemist wrote: post #39
nomel wrote: post #29

Earlier quoted context omitted.

Lowest I've used is 8 bit floats for time delays, in embedded devices.

Interesting! I have been using integers or f32 for that. What was the use case specifically? Did you write a software float for it? I remember writing a `f16` type for an IC that used that was a pain!

Tight memory constraint. I was putting configuration somewhere it shouldn't have been, but it meant we didn't need to buy an extra chip.

Yes, purely software.

Re: 4-bit floating point FP4

#73

Post by adampunk »

adrian_b wrote: post #70
adampunk wrote: post #67

Earlier quoted context omitted.

You are totally correct but I need you to recognize that "in ancient times" includes the 1990s. I am...very sorry to be the one delivering this news. It was not a pleasant realization for me, either.

A few years after 1980, especially after 1985, the computers with coprocessors like Intel 8087 or Motorola 68881 became the most numerous computers with floating-point hardware, and for them the default FP size was 80-bit. So the 1990s were long after the time when 32-bit FP numbers were normal. FP32 was revived only by GPUs, for graphic applications where precision matters much less. Already after 1974, the C progra…

Yeah. I know. I'm not disagreeing with your diagnosis, I'm just trying to gently rib you that your correction is misaimed. It's a joke, ya know?

>Single-precision computations for traditional applications are suitable only for experts, not for ordinary computer users.

Lots of ordinary computer users did compute in single precision! The reason I picked the 1990s as 'ancient' and not 1980 (when the 8087 was taped out) or 1985 (when IEEE754 was finally approved) was because those microprocessors were now in the hands of users who weren't under the supervision of 'experts'. That, along with the lack of fast 64 bit registers + the desire for high throughput at low fidelity led to a lot of 32 bit code!

And, frankly, if you want to get real technical, the ability of non-experts to program in FP in 64 bit is enforced NOT ONLY by the doubled bits but by the implicit ability (absent now in many implementations) to use the 80 bit extended precision format for intermediate calcs. It's the added bits in that format for scratch that let lots of 64 bit programs just work.

Re: 4-bit floating point FP4

#74

Post by amavect »

nivertech wrote: post #36

FP2 spec: 00 -> 0.0 01 -> 1.0 10 -> Inf 11 -> NaN or 00 -> 0.0 01 -> 1.0 10 -> Inf 11 -> -Inf

  00 ->  0.0
  01 -> +1.0
  10 ->  NaN
  11 -> -1.0
Arithmetic:

  0.0 + x = x
  NaN + x = NaN
  +1.0 + -1.0 = 0.0
  +1.0 + +1.0 = NaN
  -1.0 + -1.0 = NaN
  
  -0.0 = 0.0
  -(+1.0) = -1.0
  -(-1.0) = +1.0
  -NaN = NaN
  
  x - y = x + (-y)
  
  NaN * x = NaN
  +1.0 * x = x
  -1.0 * x = -x
  0.0 * 0.0 = 0.0
  
  /0.0 = NaN
  /+1.0 = +1.0
  /-1.0 = -1.0
  /NaN = NaN
  
  x / y = x * (/y)
More interestingly, how to implement in logic gates. Addition with a 2's complement full adder and NaN detector. Negation with a 2's complement negation circuit. Reciprocal with a 0.0 detector.

Multiplication with a unique logic circuit (use a Karnaugh map):

  (ab * cd) = (a&~b | c&~d | ~a&b&c | a&~c&d)(b & d)

Re: 4-bit floating point FP4

#75

Post by 0-_-0 »

mysterydip wrote: post #40
0-_-0 wrote: post #38

Earlier quoted context omitted.

You want to make multiplication cheap, it's not just about compression

Wouldn’t multiplication just be an 8 bit lookup table? a*b is just lut[a<<4+b]

A 256 element lookup table is much bigger than a simple multiplier

Re: 4-bit floating point FP4

#76

Post by nivertech »

amavect wrote: post #74
nivertech wrote: post #36

FP2 spec: 00 -> 0.0 01 -> 1.0 10 -> Inf 11 -> NaN or 00 -> 0.0 01 -> 1.0 10 -> Inf 11 -> -Inf

00 -> 0.0 01 -> +1.0 10 -> NaN 11 -> -1.0 Arithmetic: 0.0 + x = x NaN + x = NaN +1.0 + -1.0 = 0.0 +1.0 + +1.0 = NaN -1.0 + -1.0 = NaN -0.0 = 0.0 -(+1.0) = -1.0 -(-1.0) = +1.0 -NaN = NaN x - y = x + (-y) NaN * x = NaN +1.0 * x = x -1.0 * x = -x 0.0 * 0.0 = 0.0 /0.0 = NaN /+1.0 = +1.0 /-1.0 = -1.0 /NaN = NaN x / y = x * (/y) More interestingly, how to implement in logic gates. Addition with a 2's complement full adder…

What about comparison operators?

Re: 4-bit floating point FP4

#77

Post by saulpw »

saulpw wrote: post #46
petters wrote: post #45

Earlier quoted context omitted.

That's a good idea and it exists: https://www.johndcook.com/blog/2026/04/18/qlora/ It seems quite wastful to have two zeros when you only have 4 bits it total

OTOH, it seems quite plausible that the most important numbers to represent are: +0 -0 +1 -1 +inf -inf

I can see the most important values being:

   ± 0 (infinitesimal)
   ± 10^-2n
   ± 10^-n
   ± 1 (unity)
   ± 10^n
   ± 10^2n
   ± infinity
For fp4, this leaves 2 values. Maybe one of them should be NaN. What should the other one be?

Re: 4-bit floating point FP4

#78

Post by londons_explore »

0-_-0 wrote: post #38
teo_zero wrote: post #33

When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers. As a bonus, any operation can be replaced with a lookup into a nxn table.

You want to make multiplication cheap, it's not just about compression

Specifically, you want to choose 16 values, all of which you can multiply an activation value by using circuitry which is as small as possible.

Re: 4-bit floating point FP4

#79

Post by amavect »

nivertech wrote: post #76
amavect wrote: post #74

Earlier quoted context omitted.

00 -> 0.0 01 -> +1.0 10 -> NaN 11 -> -1.0 Arithmetic: 0.0 + x = x NaN + x = NaN +1.0 + -1.0 = 0.0 +1.0 + +1.0 = NaN -1.0 + -1.0 = NaN -0.0 = 0.0 -(+1.0) = -1.0 -(-1.0) = +1.0 -NaN = NaN x - y = x + (-y) NaN * x = NaN +1.0 * x = x -1.0 * x = -x 0.0 * 0.0 = 0.0 /0.0 = NaN /+1.0 = +1.0 /-1.0 = -1.0 /NaN = NaN x / y = x * (/y) More interestingly, how to implement in logic gates. Addition with a 2's complement full adder…

What about comparison operators?

I'll use custom notation =? ≤≥? 
  x =? x = True
  Otherwise, a =? b = False
  
  NaN ≤≥? NaN = False
  Otherwise, a ≤≥? b = a =? b
  
  -1.0 ? b = b ? b | a ≤≥? b)
In logic gates: For =?, bitwise equality. For ≤≥?, bitwise equality and a NaN detector. For 
  ab
I separate =? from ≤≥?. =? compares value, while ≤≥? compares order. NaN has no ordering, so it compares false. IEEE float only uses ≤≥? and names it ==.

Re: 4-bit floating point FP4

#80

Post by nivertech »

amavect wrote: post #79
nivertech wrote: post #76

Earlier quoted context omitted.

What about comparison operators?

I'll use custom notation =? ≤≥? x =? x = True Otherwise, a =? b = False NaN ≤≥? NaN = False Otherwise, a ≤≥? b = a =? b -1.0 ? b = b ? b | a ≤≥? b) In logic gates: For =?, bitwise equality. For ≤≥?, bitwise equality and a NaN detector. For ab I separate =? from ≤≥?. =? compares value, while ≤≥? compares order. NaN has no ordering, so it compares false. IEEE float only uses ≤≥? and names it ==.

It's better to first show truth tables, then K-maps, and only then logical formulas.

But the main question is: does this FP2 have any real applications? Maybe it could be useful when only one operand is FP2? Especially for vectorized math.

Post reply on HN