It seems like most breakthroughs I see are for efficiency? What are the most importsnt breakthroughs from the past two or three years for intelligence?
TurboQuant: Redefining AI efficiency with extreme compression
61–70 of 202 posts
Re: TurboQuant: Redefining AI efficiency with extreme compression
#62"TurboQuant proved it can quantize the key-value cache to just 3 bits without requiring training or fine-tuning and causing any compromise in model accuracy" -- what do each 3 bits correspond to? Hardly individual keys or values, since it would limit each of them to 8 different vectors.
Is the number of bits per coordinate. So, 1 bit is 2x2 grid. 3 bit is a 64 cell grid (2^3 x 2^3). Here you have a demo.
https://mesuvash.github.io/blog/2026/turboquant-interactive/
Re: TurboQuant: Redefining AI efficiency with extreme compression
#63Will this help us run models locally?
Re: TurboQuant: Redefining AI efficiency with extreme compression
#64Re: TurboQuant: Redefining AI efficiency with extreme compression
#65Someone implementing it on llamacpp already https://github.com/mudler/llama.cpp/commit/dee102db1bfd723c9...
Re: TurboQuant: Redefining AI efficiency with extreme compression
#66Earlier quoted context omitted.
https://mesuvash.github.io/blog/2026/turboquant-interactive/ has a little visualisation
I like the visualization, but I don’t understand the grid quantization. If every point is on the unit circle aren’t all the center grid cords unused?
Yeah that's odd. It seems like you'd want an n-1 dimensional grid on the surface of the unit sphere rather than an n dimensional grid within which the sphere resides.
Looking at the paper (https://arxiv.org/abs/2504.19874) they cite earlier work that does exactly that. They object that grid projection and binary search perform exceptionally poorly on the GPU.
I don't think they're using a regular grid as depicted on the linked page. Equation 4 from the paper is how they compute centroids for the MSE optimal quantizer.
Why specify MSE optimal you ask? Yeah so it turns out there's actually two quantization steps, a detail also omitted from the linked page. They apply QJL quantization to the residual of the grid quantized data.
My description is almost certainly missing key details; I'm not great at math and this is sufficiently dense to be a slog.
Re: TurboQuant: Redefining AI efficiency with extreme compression
#67Re: TurboQuant: Redefining AI efficiency with extreme compression
#68It seems like most breakthroughs I see are for efficiency? What are the most importsnt breakthroughs from the past two or three years for intelligence?
Efficiency gains can be used to make existing models more profitable, or to make new larger and more intelligent models.
Re: TurboQuant: Redefining AI efficiency with extreme compression
#69It seems like most breakthroughs I see are for efficiency? What are the most importsnt breakthroughs from the past two or three years for intelligence?
Most breakthroughs that are published are for efficiency because most breakthroughs that are published are for open source.'
All the foundation model breakthroughs are hoarded by the labs doing the pretraining. That being said, RL reasoning training is the obvious and largest breakthrough for intelligence in recent years.
Re: TurboQuant: Redefining AI efficiency with extreme compression
#70[dead]