Earlier quoted context omitted.
A model trained on BF16 that within the range of FP16 have effective bit rate of 13-bit at max (e5m7). Reasonable quantization (at 8-bit) gets you weight error (i.e. L2 distance on weights) down to I think there is a line somewhere between 4-bit to 8-bit that will hurt performance (for both diffusion models and LLM). But I doubt the line is between 8-bit to 13-bit. (Another case in point: you can use generic lossless…
I buy these kinds of arguments but the moment a company or NeurIPS researcher claims even a "tiny" bit of loss happens, I become suspicious. I don't buy most claims of "we get 99%+ of the performance" made in practice. But yes, I do believe that we will find proper lossless quants, and eventually (for real this time) get "only a little bit of loss" quants, but I don't think that the current 8 bits are there yet. Also…
If we start from peering into quantization, we can show it is by definition lossy, unless every term had no significant bits past the quantization amount.
so our lower bound must that 0.03% error mentioned above.