Excellent. For me, six bits of mantissa in fixed point is perfectly adequate
Standardizing next-generation narrow precision data formats for AI
21–30 of 54 posts
Re: Standardizing next-generation narrow precision data formats for AI
#22Graphcore have been doing some good research in the area: https://twitter.com/sergiopprz/status/1708827369031516169
Re: Standardizing next-generation narrow precision data formats for AI
#23Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…
Thanks - interesting. I wish > Integer data types use a 2’s complement encoding, but the maximum negative representation (−2) may be left unused to maintain symmetry between the maximum positive and negative representations and avoid introducing a negative bias. ... the maximum negative representation was used for a NAN. IDK why and how it is that we all agree that NAN-s are useful for floats (and they are super usef…
> IDK why and how it is that we all agree that NAN-s are useful for floats (and they are super useful), but very few think the same for integers??
Because making an integer bit pattern act as a NaN would require specific semantics (e.g. NaN + X = NaN; NaN != NaN) which are difficult to implement efficiently in hardware. These properties would also potentially rule out some arithmetic optimizations which are currently possible.
Re: Standardizing next-generation narrow precision data formats for AI
#24Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…
For us less technical folks (in this field), what’s the big take away here / why does this matter / why should we be excited?
[deleted]
Re: Standardizing next-generation narrow precision data formats for AI
#25Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, builds on a foundation of years of design space exploration and res…
For us less technical folks (in this field), what’s the big take away here / why does this matter / why should we be excited?
On most hardware, handwritten math is required for all the nonstandard formats, e.g. for quantized int-8 https://github.com/karpathy/llama2.c/blob/master/runq.c#L317
Integer quantization doesn't typically just round, it has scaling and other factors in blocks so it's not just a question of manipulating int8's. And the FP16/FP8 are not supported by most processors so need their own custom routines as well. It would be great if you could just write code that operates with intrinsics on the quanitzed types.
Re: Standardizing next-generation narrow precision data formats for AI
#26What would be the most practical range for a hypothetical 1-bit floating point type (float1_t)? Zero and one? Zero and infinity? NaN and infinity?
Re: Standardizing next-generation narrow precision data formats for AI
#27Cool. The way current hardware handles very low precision is quite inefficient.
Indeed, reading the QLoRa paper, 4 bit quantised data is converted to 16 bit floats (usually BFloat16) for calculations, then converted back again. I suppose this standard allows for smaller data types to be supported by the hardware instructions.
Re: Standardizing next-generation narrow precision data formats for AI
#28Huh, for FP4 just E2M1 with no E3M0? I've seen a paper in the past that went so heavy on exponent it was skipping every other power of two, so I would have thought the demand was there.
Oddly they do have E8M0.
Re: Standardizing next-generation narrow precision data formats for AI
#29Huh, for FP4 just E2M1 with no E3M0? I've seen a paper in the past that went so heavy on exponent it was skipping every other power of two, so I would have thought the demand was there. Oddly they do have E8M0.
E3M0 was the format I was most excited to see here, but I guess not. E8M0 makes sense because of the relationship to E8M23 (float32) and E8M7 (bfloat16). Nvidia has their own E8M12 format that uses the exponent logic of float32 and the mantissa logic of float16, allowing you to multiply 2x more numbers at a time in E8M12 as E8M23 without adding more hardware or resorting to a narrower exponent.
Re: Standardizing next-generation narrow precision data formats for AI
#30Any idea how these formats compare to POSITs for AI computations? Also: It's not cool IMHO that they have two distinct formats for FP8.
The promises of POSITs don't seem to hold water in terms of their application-level benefits, n-bit posits need equivalent-sized hardware to 2n-bit floats, and the numerical analysis on them is hell. All in all, they were an interesting thought experiment. Compressed/quantized storage of floating point numbers seems to just be better.