Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Standardizing next-generation narrow precision data formats for AI

opencompute.org

Post reply on HN
Print view View original

31–40 of 54 posts

Re: Standardizing next-generation narrow precision data formats for AI

#31

Post by Dylan16807 »

cpeterso wrote: post #26

What would be the most practical range for a hypothetical 1-bit floating point type (float1_t)? Zero and one? Zero and infinity? NaN and infinity?

As far as successful calculations go, NaN is useless and infinity is next to useless.

And any nonzero finite number might as well be 1 to make it easier to think about.

0 and 1 would work, as long as you have a final scaling factor per node. -1 and 1 should work too.

Also the entire concept of a floating point is weak enough at 3-4 bits and collapses entirely when you have 1 or 2.

Re: Standardizing next-generation narrow precision data formats for AI

#32

Post by invalidator »

cpeterso wrote: post #26

What would be the most practical range for a hypothetical 1-bit floating point type (float1_t)? Zero and one? Zero and infinity? NaN and infinity?

I'd say a type which doesn't have both a significand and an exponent isn't a float at all. How can you have a floating point with no ability to move the point?

To have something IEEE754-like you'd need a sign, significand, and exponent. That gets you +/-(0, 1, Inf, Nan). Add another bit to the exponent if you want non-integer values. That's the minimum for what I'd call a float.

If you really want to strip it down to one bit, you should choose which of those bits to keep. Significand gives you (0,1), which is the same as a uint1_t. Exponent arguably gives you (0, Inf). Sign gives you (-1,1).

Of those, I'd say the sign-bit, giving -1 and 1, is maybe the most practical, but it's not a complete float.

Re: Standardizing next-generation narrow precision data formats for AI

#33

Post by dragonwriter »

samspenc wrote: post #5

Interesting to see Nvidia here - I would assume they have the most to lose from an open consortium like this. Or do they think they will come out ahead even if there is an open standard like this.

They have the most to lose by their being a standard that they don’t participate in, though, yes, no standard at all is better for them than a standard they do participate in.

Re: Standardizing next-generation narrow precision data formats for AI

#35

Post by Dylan16807 »

pclmulqdq wrote: post #30
phkahler wrote: post #8

Any idea how these formats compare to POSITs for AI computations? Also: It's not cool IMHO that they have two distinct formats for FP8.

The promises of POSITs don't seem to hold water in terms of their application-level benefits, n-bit posits need equivalent-sized hardware to 2n-bit floats, and the numerical analysis on them is hell. All in all, they were an interesting thought experiment. Compressed/quantized storage of floating point numbers seems to just be better.

> n-bit posits need equivalent-sized hardware to 2n-bit floats

Not in general. Every 8 bit posit with es=1 can be decoded to a 10 bit E5M4 float. Every 8 bit posit with es=2 can be decoded to E6M3, and almost all fit into E5M3.

Though overall I'm not sure if posits are particularly useful here. Posits give you more precision near 1.0, and they give you better range for outliers. The latter almost certainly helps, but I don't know about the former.

Saving 1 bit wouldn't hurt though.

Re: Standardizing next-generation narrow precision data formats for AI

#36

Post by buildbot »

Dylan16807 wrote: post #28

Huh, for FP4 just E2M1 with no E3M0? I've seen a paper in the past that went so heavy on exponent it was skipping every other power of two, so I would have thought the demand was there. Oddly they do have E8M0.

Point of clarification - there is no E8M0 direct datatype (unless I misunderstand something!) E8M0 is only used for the scaling of exponents in the block - there is 8 bits of scale per block.

Re: Standardizing next-generation narrow precision data formats for AI

#37

Post by buildbot »

pclmulqdq wrote: post #29
Dylan16807 wrote: post #28

Huh, for FP4 just E2M1 with no E3M0? I've seen a paper in the past that went so heavy on exponent it was skipping every other power of two, so I would have thought the demand was there. Oddly they do have E8M0.

E3M0 was the format I was most excited to see here, but I guess not. E8M0 makes sense because of the relationship to E8M23 (float32) and E8M7 (bfloat16). Nvidia has their own E8M12 format that uses the exponent logic of float32 and the mantissa logic of float16, allowing you to multiply 2x more numbers at a time in E8M12 as E8M23 without adding more hardware or resorting to a narrower exponent.

Copy my comment here too - Point of clarification - there is no E8M0 direct datatype (unless I misunderstand something!) E8M0 is only used for the scaling of exponents in the block - there is 8 bits of scale per block.

Re: Standardizing next-generation narrow precision data formats for AI

#38

Post by buildbot »

kristianp wrote: post #27
brucethemoose2 wrote: post #3

Cool. The way current hardware handles very low precision is quite inefficient.

Indeed, reading the QLoRa paper, 4 bit quantised data is converted to 16 bit floats (usually BFloat16) for calculations, then converted back again. I suppose this standard allows for smaller data types to be supported by the hardware instructions.

Exactly!

Re: Standardizing next-generation narrow precision data formats for AI

#39

Post by notnullorvoid »

Floats seem like a pretty bad data format for ML (Also in general). Infinity, NaN, -0 all useless. NaNs in particular have multiple binary representations so waste extra space.

Not sure why more effort isn't being put toward Posits, or thinking up a different format for ML specifically.

Re: Standardizing next-generation narrow precision data formats for AI

#40

Post by jcranmer »

notnullorvoid wrote: post #39

Floats seem like a pretty bad data format for ML (Also in general). Infinity, NaN, -0 all useless. NaNs in particular have multiple binary representations so waste extra space. Not sure why more effort isn't being put toward Posits, or thinking up a different format for ML specifically.

Well, if you actually read the document, that's not how NaN is encoded in these data types--these are not IEEE 754-compliant encodings.

As for why not posits, I'm not entirely sure, but the "variably-encoded exponent width" nature of posits likely makes several details of their construction in hardware more difficult than things that have fixed exponent and mantissa widths. Although by the time you're talking about 8-bit datatypes, implementing operations by lookup table starts to look appealing.

Post reply on HN