Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Standardizing next-generation narrow precision data formats for AI

opencompute.org

Post reply on HN
Print view View original

11–20 of 54 posts

Re: Standardizing next-generation narrow precision data formats for AI

#11

Post by superkuh »

opcode84 wrote: post #2

Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…

Actual article as text for those like myself getting completely blocked by the broken captcha wall. I must have hit the "prove you're not a robot" check-box 20 times.

https://web.archive.org/web/20231018183224/https://www.openc...

Re: Standardizing next-generation narrow precision data formats for AI

#12

Post by redox99 »

yardstick wrote: post #10
brucethemoose2 wrote: post #3

Cool. The way current hardware handles very low precision is quite inefficient.

Stupid question but will people start pushing for more precision once the cost of memory and compute falls more? Or by the nature of these data types there will never really be a need for more bits?

If memory and compute prices fall, people will just train bigger models.

Re: Standardizing next-generation narrow precision data formats for AI

#13

Post by dist-epoch »

yardstick wrote: post #10
brucethemoose2 wrote: post #3

Cool. The way current hardware handles very low precision is quite inefficient.

Stupid question but will people start pushing for more precision once the cost of memory and compute falls more? Or by the nature of these data types there will never really be a need for more bits?

Both. You could either go for more precision or go for bigger models.

Re: Standardizing next-generation narrow precision data formats for AI

#14

Post by cyrillite »

opcode84 wrote: post #2

Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…

For us less technical folks (in this field), what’s the big take away here / why does this matter / why should we be excited?

Re: Standardizing next-generation narrow precision data formats for AI

#15

Post by buildbot »

cyrillite wrote: post #14
opcode84 wrote: post #2

Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…

For us less technical folks (in this field), what’s the big take away here / why does this matter / why should we be excited?

Typically, you need to use some tricks for pre-training in lower precision (finetuning seems to work at low precision), with FP16 you need loss scaling for example. With MX, you can train in 6 bits of precision without any tricks, and hit the same loss as FP32.

Re: Standardizing next-generation narrow precision data formats for AI

#16

Post by buildbot »

SR2Z wrote: post #7
samspenc wrote: post #5

Interesting to see Nvidia here - I would assume they have the most to lose from an open consortium like this. Or do they think they will come out ahead even if there is an open standard like this.

I think that while Nvidia has a near-monopoly on training, the inference side of things is much more multiplatform. That's probably why.

This is for both training and inference, for what it is worth.

Re: Standardizing next-generation narrow precision data formats for AI

#17

Post by brucethemoose2 »

yardstick wrote: post #10
brucethemoose2 wrote: post #3

Cool. The way current hardware handles very low precision is quite inefficient.

Stupid question but will people start pushing for more precision once the cost of memory and compute falls more? Or by the nature of these data types there will never really be a need for more bits?

Different types of models seem to have different "sweet spots."

For example, current transformers llms seem to like 4-6 bits with smart quantization, with good performance at 3-4 bits with extremely aggressive quantization methods (like good use of sparsity and profiling inference on useful data).

The Stable Diffusion unet doesn't like 8 bit without some changes, the vae barely even likes fp16.

So to answer your question, some quantization is "free" and theres no reason not to use it, but sometimes its very lossy and a serious compromise that would not be taken with more compute/ram.

Also, sometimes there is compute overhead that makes quantization inferencr/training slower. Sometimes the reduced model weights size makes passes faster due to a memory bandwidth bottleneck. It just depends.

Re: Standardizing next-generation narrow precision data formats for AI

#20

Post by imjonse »

cyrillite wrote: post #14
opcode84 wrote: post #2

Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…

For us less technical folks (in this field), what’s the big take away here / why does this matter / why should we be excited?

Future hardware implementations for these <8bit data types will result in much larger (number of parameters) models fitting in the same memory. Unless they are standardized, each vendor and software framework will have their own slightly different approach.
Post reply on HN