Standardizing next-generation narrow precision data formats for AI
1–10 of 54 posts
Re: Standardizing next-generation narrow precision data formats for AI
#2Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and research. MX enhances the robustness and ease-of-use of existing 8-bit formats such as FP8 and INT8, thus lowering the barrier for broader adoption of single digit bit training and inference.
Spec: https://www.opencompute.org/documents/ocp-microscaling-forma...
Whitepaper: https://arxiv.org/abs/2310.10537
Re: Standardizing next-generation narrow precision data formats for AI
#3Cool. The way current hardware handles very low precision is quite inefficient.
Re: Standardizing next-generation narrow precision data formats for AI
#4Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…
Thanks - interesting. I wish
> Integer data types use a 2’s complement encoding, but the maximum negative representation (−2) may be left unused to maintain symmetry between the maximum positive and negative representations and avoid introducing a negative bias.
... the maximum negative representation was used for a NAN. IDK why and how it is that we all agree that NAN-s are useful for floats (and they are super useful), but very few think the same for integers??
Re: Standardizing next-generation narrow precision data formats for AI
#5Interesting to see Nvidia here - I would assume they have the most to lose from an open consortium like this. Or do they think they will come out ahead even if there is an open standard like this.
Re: Standardizing next-generation narrow precision data formats for AI
#6Nice, standardization is essential. Without it things could get overly complicated and hard to develop technology to support the consumption of data outside of the convention. Although I would assume that there will be updates continually to the model.
Re: Standardizing next-generation narrow precision data formats for AI
#7Interesting to see Nvidia here - I would assume they have the most to lose from an open consortium like this. Or do they think they will come out ahead even if there is an open standard like this.
I think that while Nvidia has a near-monopoly on training, the inference side of things is much more multiplatform. That's probably why.
Re: Standardizing next-generation narrow precision data formats for AI
#8Any idea how these formats compare to POSITs for AI computations?
Also: It's not cool IMHO that they have two distinct formats for FP8.
Re: Standardizing next-generation narrow precision data formats for AI
#9Interesting to see Nvidia here - I would assume they have the most to lose from an open consortium like this. Or do they think they will come out ahead even if there is an open standard like this.
Back in the day, NVidia's OpenCL implementation beat AMD's OpenCL implementation hands down. Even die-hard AMD fans who were determined to tough it out would eventually sniff out that the grass was greener on the other side and switch teams. It was excellent advertising.
Re: Standardizing next-generation narrow precision data formats for AI
#10Cool. The way current hardware handles very low precision is quite inefficient.
Stupid question but will people start pushing for more precision once the cost of memory and compute falls more? Or by the nature of these data types there will never really be a need for more bits?