Viewing profile — opcode84

For narrow-precision formats to be practical in large-scale pretraining, they must ensure both model accuracy and stable convergence. To assess the viability of 4-bit precision in …

A version of the 12B Hybrid Mamba-Transformer model was initially trained with 8-bit precision—FP8, which has been shown in previous studies to closely match 16-bit precision, and …

Introducing NVFP4 for Efficient and Accurate Low-Precision Inference

The formats supported in the tutorial are the OCP microscaling formats, including mxfp4 and mxfp8, as well as NVIDIA’s nvfp4 format. These matrix multiplications are accelerated by…

From the article: "NVIDIA Blackwell introduces revolutionary block-scaled floating point formats, including the Open Computing Project’s microscaling formats, which Triton now unlo…

https://arxiv.org/pdf/2310.10537.pdf Discussed in a previous post https://news.ycombinator.com/item?id=37930663

The Microsoft chip has MX data types: https://news.ycombinator.com/item?id=37930663 https://arxiv.org/abs/2310.10537

Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizi…