Untitled topic

SOTA quantization aware training (QAT) techniques summary.

1. Bitnet weights are one of three digits (-1, 0, 1). This is the OG 1.58 bit network.

2. Sherry quantization constrains every 3 out of 4 Bitnet weights to non zero for SIMD optimization. Do the math and you'll see this is a 1.25 bit network.

3. Tequila quantization handles excess zeros due to gradient flow issues.

4. Fairy quantization is Bitnet plus an imaginary number {±i}. So it's a 2-Bit network, not a ternary network.