Standardizing next-generation narrow precision data formats for AI • Hacker News Board

Re: Standardizing next-generation narrow precision data formats for AI

#51

Post by Y_Y » Thu, Oct 19, 2023, 10:40 PM UTC

invalidator wrote: post #32

cpeterso wrote: post #26

What would be the most practical range for a hypothetical 1-bit floating point type (float1_t)? Zero and one? Zero and infinity? NaN and infinity?

I'd say a type which doesn't have both a significand and an exponent isn't a float at all. How can you have a floating point with no ability to move the point? To have something IEEE754-like you'd need a sign, significand, and exponent. That gets you +/-(0, 1, Inf, Nan). Add another bit to the exponent if you want non-integer values. That's the minimum for what I'd call a float. If you really want to strip it down to…

Three-bit floats have been the subject almost serious research, http://tom7.org/nand/nand.pdf

Re: Standardizing next-generation narrow precision data formats for AI

#52

Post by buildbot » Fri, Oct 20, 2023, 5:29 AM UTC

pclmulqdq wrote: post #48

buildbot wrote: post #37

Earlier quoted context omitted.

Copy my comment here too - Point of clarification - there is no E8M0 direct datatype (unless I misunderstand something!) E8M0 is only used for the scaling of exponents in the block - there is 8 bits of scale per block.

I think you're right. In general, storage and operating formats seem to be decoupling for AI/ML. Nvidia's E8M12 is also a format specifically for operators - they expect you to store FP32 when you operate in E8M12. Storage is almost always in power-of-2 sizes.

I would hope so ;)

Re: Standardizing next-generation narrow precision data formats for AI

#53

Post by buildbot » Fri, Oct 20, 2023, 9:04 PM UTC

bmh wrote: post #44

It's interesting that the standard "K" (number of elements with a shared scale) is 32. That seems to imply that the neural network will somehow learn to group weights at those 32-element boundaries. Does anybody understand how that works? I mean, what is the mechanism that naturally causes the model to group weight scales into those K-element clusters?

There is no mechanism per-say, it's more of a bit space vs quality issue. You could think of MX4 with an 8 bit exponent scale as a 12 bit number if the block size is one, "MX12" with E10M1. You can share the scale with some error per element in a block, with that error going up as you increase the size of the block. As the block size is increased, the effective size per element goes down and the hardware implementation gets smaller/cheaper.

Re: Standardizing next-generation narrow precision data formats for AI

#54

Post by buildbot » Fri, Oct 20, 2023, 9:08 PM UTC

larodi wrote: post #46

Sparkyte wrote: post #6

Nice, standardization is essential. Without it things could get overly complicated and hard to develop technology to support the consumption of data outside of the convention. Although I would assume that there will be updates continually to the model.

this all is based on the voluntary work of many researchers in academia and opensource, and now these big companies are 'standardizing' stuff they did not invent, but was invented despite them going in other market directions..

Not true in this case, it was developed in the industry. https://azure.microsoft.com/en-us/blog/fostering-ai-infrastr...

"Building on years of design space exploration and research at Microsoft, Microscaling technology enables sub 8-bit formats while also enhancing the strength and ease-of-use of existing 8-bit formats such as FP8 and INT8. These advancements also help contribute to broader sustainability goals like reducing the environmental impact of AI technologies as demand continues to grow by improving the energy efficiency of AI in datacenters as well as on many AI endpoints."

Of course, it was not the first block floating point, those have been around since the 1963! https://en.wikipedia.org/wiki/Block_floating_point.