Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Standardizing next-generation narrow precision data formats for AI

opencompute.org

Post reply on HN
Print view View original

41–50 of 54 posts

Re: Standardizing next-generation narrow precision data formats for AI

#41

Post by notnullorvoid »

jcranmer wrote: post #40
notnullorvoid wrote: post #39

Floats seem like a pretty bad data format for ML (Also in general). Infinity, NaN, -0 all useless. NaNs in particular have multiple binary representations so waste extra space. Not sure why more effort isn't being put toward Posits, or thinking up a different format for ML specifically.

Well, if you actually read the document, that's not how NaN is encoded in these data types--these are not IEEE 754-compliant encodings. As for why not posits, I'm not entirely sure, but the "variably-encoded exponent width" nature of posits likely makes several details of their construction in hardware more difficult than things that have fixed exponent and mantissa widths. Although by the time you're talking about 8…

Ahh okay wasn't aware the encodings were different, after taking a glance at spec it seems much better than what I pessimistically envisioned.

2 NaNs, and -0 still seem like a bad call (-0 could have been the singular NaN), but I guess I understand maybe why they don't want to deviate too much from IEEE754 floats.

Re: Standardizing next-generation narrow precision data formats for AI

#42

Post by quotemstr »

notnullorvoid wrote: post #39

Floats seem like a pretty bad data format for ML (Also in general). Infinity, NaN, -0 all useless. NaNs in particular have multiple binary representations so waste extra space. Not sure why more effort isn't being put toward Posits, or thinking up a different format for ML specifically.

Meanwhile, people look at you like you have two heads if you suggest using anything other than "float" in certain contexts. "How can you do subpixel positioning without float?!?!?! This is unacceptable!" "You can use 1/1000 of a dp to express fractional pix..." "FLOATS. FLOATS GIVE US FRACTIONS. WHY DO YOU HATE OUR CUSTOMERS?".

IEE754 floats should never have been a primitive in any programming language. Library type only.

Re: Standardizing next-generation narrow precision data formats for AI

#43

Post by sanxiyn »

opcode84 wrote: post #2

Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…

This reminds me a lot of Nervana Systems' Flexpoint.

Re: Standardizing next-generation narrow precision data formats for AI

#44

Post by bmh »

It's interesting that the standard "K" (number of elements with a shared scale) is 32. That seems to imply that the neural network will somehow learn to group weights at those 32-element boundaries. Does anybody understand how that works? I mean, what is the mechanism that naturally causes the model to group weight scales into those K-element clusters?

Re: Standardizing next-generation narrow precision data formats for AI

#45

Post by ljosifov »

duskwuff wrote: post #23
ljosifov wrote: post #4

Earlier quoted context omitted.

Thanks - interesting. I wish > Integer data types use a 2’s complement encoding, but the maximum negative representation (−2) may be left unused to maintain symmetry between the maximum positive and negative representations and avoid introducing a negative bias. ... the maximum negative representation was used for a NAN. IDK why and how it is that we all agree that NAN-s are useful for floats (and they are super usef…

> IDK why and how it is that we all agree that NAN-s are useful for floats (and they are super useful), but very few think the same for integers?? Because making an integer bit pattern act as a NaN would require specific semantics (e.g. NaN + X = NaN; NaN != NaN) which are difficult to implement efficiently in hardware. These properties would also potentially rule out some arithmetic optimizations which are currently…

Yep - agreed that there will be a price to be paid. Do you have any inkling of how costly in h/w would it be? Maybe compared to the similar in floats? For floats it's been decided that paying the price was worth it, it seems.

Re: Standardizing next-generation narrow precision data formats for AI

#46

Post by larodi »

Sparkyte wrote: post #6

Nice, standardization is essential. Without it things could get overly complicated and hard to develop technology to support the consumption of data outside of the convention. Although I would assume that there will be updates continually to the model.

this all is based on the voluntary work of many researchers in academia and opensource, and now these big companies are 'standardizing' stuff they did not invent, but was invented despite them going in other market directions..

Re: Standardizing next-generation narrow precision data formats for AI

#47

Post by pclmulqdq »

Dylan16807 wrote: post #35
pclmulqdq wrote: post #30

Earlier quoted context omitted.

The promises of POSITs don't seem to hold water in terms of their application-level benefits, n-bit posits need equivalent-sized hardware to 2n-bit floats, and the numerical analysis on them is hell. All in all, they were an interesting thought experiment. Compressed/quantized storage of floating point numbers seems to just be better.

> n-bit posits need equivalent-sized hardware to 2n-bit floats Not in general. Every 8 bit posit with es=1 can be decoded to a 10 bit E5M4 float. Every 8 bit posit with es=2 can be decoded to E6M3, and almost all fit into E5M3. Though overall I'm not sure if posits are particularly useful here. Posits give you more precision near 1.0, and they give you better range for outliers. The latter almost certainly helps, but…

If you think of posits as a storage format, it's true that you only need an extra posit->float decoder, but you need bfloat16 to exactly decode all 8-bit posits into a single, common FP format.

In terms of hardware to operate directly on posits, things get ugly. Floats are actually relatively easy to work with due to the normalized storage format and fixed precision across the range of numbers, while posits don't have that.

Re: Standardizing next-generation narrow precision data formats for AI

#48

Post by pclmulqdq »

buildbot wrote: post #37
pclmulqdq wrote: post #29

Earlier quoted context omitted.

E3M0 was the format I was most excited to see here, but I guess not. E8M0 makes sense because of the relationship to E8M23 (float32) and E8M7 (bfloat16). Nvidia has their own E8M12 format that uses the exponent logic of float32 and the mantissa logic of float16, allowing you to multiply 2x more numbers at a time in E8M12 as E8M23 without adding more hardware or resorting to a narrower exponent.

Copy my comment here too - Point of clarification - there is no E8M0 direct datatype (unless I misunderstand something!) E8M0 is only used for the scaling of exponents in the block - there is 8 bits of scale per block.

I think you're right. In general, storage and operating formats seem to be decoupling for AI/ML.

Nvidia's E8M12 is also a format specifically for operators - they expect you to store FP32 when you operate in E8M12. Storage is almost always in power-of-2 sizes.

Re: Standardizing next-generation narrow precision data formats for AI

#49

Post by Dylan16807 »

pclmulqdq wrote: post #47
Dylan16807 wrote: post #35

Earlier quoted context omitted.

> n-bit posits need equivalent-sized hardware to 2n-bit floats Not in general. Every 8 bit posit with es=1 can be decoded to a 10 bit E5M4 float. Every 8 bit posit with es=2 can be decoded to E6M3, and almost all fit into E5M3. Though overall I'm not sure if posits are particularly useful here. Posits give you more precision near 1.0, and they give you better range for outliers. The latter almost certainly helps, but…

If you think of posits as a storage format, it's true that you only need an extra posit->float decoder, but you need bfloat16 to exactly decode all 8-bit posits into a single, common FP format. In terms of hardware to operate directly on posits, things get ugly. Floats are actually relatively easy to work with due to the normalized storage format and fixed precision across the range of numbers, while posits don't hav…

> If you think of posits as a storage format, it's true that you only need an extra posit->float decoder, but you need bfloat16 to exactly decode all 8-bit posits into a single, common FP format.

But you're not limited to powers of two. If you're decoding to 16 bit floats then you can use posits up to 13 bits wide.

> In terms of hardware to operate directly on posits, things get ugly. Floats are actually relatively easy to work with due to the normalized storage format and fixed precision across the range of numbers, while posits don't have that.

But is that extra shifting bigger than the multiplier unit? It's hard for me to see it growing the circuit that much.

Re: Standardizing next-generation narrow precision data formats for AI

#50

Post by duskwuff »

ljosifov wrote: post #45
duskwuff wrote: post #23

Earlier quoted context omitted.

> IDK why and how it is that we all agree that NAN-s are useful for floats (and they are super useful), but very few think the same for integers?? Because making an integer bit pattern act as a NaN would require specific semantics (e.g. NaN + X = NaN; NaN != NaN) which are difficult to implement efficiently in hardware. These properties would also potentially rule out some arithmetic optimizations which are currently…

Yep - agreed that there will be a price to be paid. Do you have any inkling of how costly in h/w would it be? Maybe compared to the similar in floats? For floats it's been decided that paying the price was worth it, it seems.

> Do you have any inkling of how costly in h/w would it be?

For something like addition, a ripple-carry adder is ~5 gates per bit. To check for NaN on input, you'd need a wide AND/OR on each input (~N log N gates per bit per input, so ~4 gates per bit for a 64-bit adder), a multiplexer on the output (~3 gates per bit), and a bunch of fan-in/out for the "is this NaN" signal. That'd likely more than double the size of the cell.

Subtraction makes that even more awkward. With 2's complement, an adder can also perform subtraction by inverting the second operand and carrying in a 1. This trick stops working if one of your bit patterns is a special value, so you either have to add even more logic to specify that NaN is inverted, or duplicate the whole mess for subtraction.

You'd also have to add a bunch of completely new hardware to distinguish between e.g. "X is bitwise equal to Y" and "X is numerically equal to Y" in equality tests, because NaN != NaN. It's hard to speculate how expensive that would be, but it certainly wouldn't be trivial.

> For floats it's been decided that paying the price was worth it, it seems.

That's a bit of an oversimplification. I'd say that it's more that:

1) Floating-point arithmetic is already fairly complex; even if you didn't handle the special values it'd still require a lot more logic than integer math.

2) Handling special values like infinities and NaN was simply part of the "spec" for floating-point math. It wouldn't have been considered fit for purpose without those features.

Post reply on HN