TurboQuant: Redefining AI efficiency with extreme compression

Post by rohansood15 » Wed, Mar 25, 2026, 4:32 PM UTC

photon_lines wrote: post #100

gavinray wrote: post #80

Can someone ELI5 these two concepts please, which make no sense to me: > "TurboQuant starts by randomly rotating the data vectors. This clever step simplifies the data's geometry" I don't understand how taking a series of data and applying a random rotation could mathemetically lead every time to "simpler" geometry. If I throw a bunch of shapes on the ground, tightly packed and touching each other, then rotate all of…

The whole goal of quantisation is to put the data into 'bins' so that it can easily be 'packed' so that you can represent it using less bits (less information). You can think of it like rounding essentially (3.14159 -> 3). Now, sometimes within data, the distribution will be non-ideal for separating it out into bins (let's say that our rounding rules are simple -- we simply use a floor function so 2.45 maps to 2 and…

Post by rohansood15 » Wed, Mar 25, 2026, 4:38 PM UTC

dryarzeg wrote: post #20

maurelius2 wrote: post #14

I'm somewhat at a loss here other than understanding the fundamentals. Can someone tell me how the compression impact performance?

If in short, for many inference tasks the bottleneck is memory bandwidth. Suppose you have a machine with a memory bandwidth of 256 GB/s, and let's say you want to do inference for 4B model (model with 4 billion parameters). If you will load the model in BF16 format (16 bits), each forward pass (i.e. each token generated) will require roughly ~8 GB of memory bandwidth. So, 256/8 = 32 t/s, and that's the generation sp…

Post by nico » Wed, Mar 25, 2026, 5:03 PM UTC

photon_lines wrote: post #100

gavinray wrote: post #80

Can someone ELI5 these two concepts please, which make no sense to me: > "TurboQuant starts by randomly rotating the data vectors. This clever step simplifies the data's geometry" I don't understand how taking a series of data and applying a random rotation could mathemetically lead every time to "simpler" geometry. If I throw a bunch of shapes on the ground, tightly packed and touching each other, then rotate all of…

The whole goal of quantisation is to put the data into 'bins' so that it can easily be 'packed' so that you can represent it using less bits (less information). You can think of it like rounding essentially (3.14159 -> 3). Now, sometimes within data, the distribution will be non-ideal for separating it out into bins (let's say that our rounding rules are simple -- we simply use a floor function so 2.45 maps to 2 and…

Post by tripplyons » Wed, Mar 25, 2026, 5:20 PM UTC

eecc wrote: post #57

amitport wrote: post #15

This is a great development for KV cache compression. I did notice a missing citation in the related works regarding the core mathematical mechanism, though. The foundational technique of applying a geometric rotation prior to extreme quantization, specifically for managing the high-dimensional geometry and enabling proper bias correction, was introduced in our NeurIPS 2021 paper, "DRIVE" ( https://proceedings.neurip…

Pardon my simplistic question, but when you mean rotation you’re essentially talking about diagonalization aren’t you? So storing the diagonal as a matrix and the new bases is more compact?

Post by tripplyons » Wed, Mar 25, 2026, 5:24 PM UTC

busfahrer wrote: post #46

amitport wrote: post #15

This is a great development for KV cache compression. I did notice a missing citation in the related works regarding the core mathematical mechanism, though. The foundational technique of applying a geometric rotation prior to extreme quantization, specifically for managing the high-dimensional geometry and enabling proper bias correction, was introduced in our NeurIPS 2021 paper, "DRIVE" ( https://proceedings.neurip…

I just today learned about Multi-Head Latent Attention, which is also sort of a way of compressing the KV cache. Can someone explain how this new development relates to MHLA?