Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant: Redefining AI efficiency with extreme compression

research.google

Post reply on HN
Print view View original

51–60 of 202 posts

Re: TurboQuant: Redefining AI efficiency with extreme compression

#52

Post by jbellis »

macleginn wrote: post #48

"TurboQuant proved it can quantize the key-value cache to just 3 bits without requiring training or fine-tuning and causing any compromise in model accuracy" -- what do each 3 bits correspond to? Hardly individual keys or values, since it would limit each of them to 8 different vectors.

The explanation is terrible, but it's clear that it's not actually lossless.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#53

Post by pstoll »

Maxious wrote: post #4
bluequbit wrote: post #2

I did not understand what polarQuant is. Is is something like pattern based compression where the algorithm finds repeating patterns and creates an index of those common symbols or numbers?

https://mesuvash.github.io/blog/2026/turboquant-interactive/ has a little visualisation

Good post but link at the end is broken.

“”” For the full technical explanation with equations, proofs, and PyTorch pseudocode, see the companion post: TurboQuant: Near-Optimal Vector Quantization Without Looking at Your Data.“

Re: TurboQuant: Redefining AI efficiency with extreme compression

#54

Post by yorwba »

busfahrer wrote: post #46
amitport wrote: post #15

This is a great development for KV cache compression. I did notice a missing citation in the related works regarding the core mathematical mechanism, though. The foundational technique of applying a geometric rotation prior to extreme quantization, specifically for managing the high-dimensional geometry and enabling proper bias correction, was introduced in our NeurIPS 2021 paper, "DRIVE" ( https://proceedings.neurip…

I just today learned about Multi-Head Latent Attention, which is also sort of a way of compressing the KV cache. Can someone explain how this new development relates to MHLA?

Multi-Head Latent attention is a redesigned attention mechanism that produces lower-dimensional KV-cache entries. Vector quantization can store KV-cache entries using a small number of bits per dimension while ensuring that the resulting attention scores don't change too much. So MLA needs to be part of the model from the beginning of training, whereas VQ can be retrofitted afterwards, and you could also combine the two.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#55

Post by x_may »

ssijak wrote: post #51

For my grug brain can somebody translate this to ELIgrug terms? Does this mean I would be able to run 500b model on my 48gb macbook without loosing quality?

KV cache compression, so how much memory the model needs to use for extending its context. Does not affect the weight size.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#57

Post by eecc »

amitport wrote: post #15

This is a great development for KV cache compression. I did notice a missing citation in the related works regarding the core mathematical mechanism, though. The foundational technique of applying a geometric rotation prior to extreme quantization, specifically for managing the high-dimensional geometry and enabling proper bias correction, was introduced in our NeurIPS 2021 paper, "DRIVE" ( https://proceedings.neurip…

Pardon my simplistic question, but when you mean rotation you’re essentially talking about diagonalization aren’t you?

So storing the diagonal as a matrix and the new bases is more compact?

Post reply on HN