Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

TurboQuant: Redefining AI efficiency with extreme compression

research.google

Post reply on HN
Print view View original

191–200 of 202 posts

Re: TurboQuant: Redefining AI efficiency with extreme compression

#191

Post by redanddead »

gavinray wrote: post #80

Can someone ELI5 these two concepts please, which make no sense to me: > "TurboQuant starts by randomly rotating the data vectors. This clever step simplifies the data's geometry" I don't understand how taking a series of data and applying a random rotation could mathemetically lead every time to "simpler" geometry. If I throw a bunch of shapes on the ground, tightly packed and touching each other, then rotate all of…

AI and graphics are matrices

Matrices are numbers [x,y,z]

GPUs are matrix processing units

Models are big matrices, we quantize them to make them small. That is lossy. Makes AI dumber the harder you quantize but lets you run inference with lesser hardware

What if you could quantize less destructively/lossy? You could make a model way smaller or make much bigger models that run on less RAM

That is what they achieved here. They're not saying that multiplying the matrices with scalars up or down helps. They're saying that by mutating and transforming the matrix with a function (ie. rotating the dimensions by the same "random" rotation) you have matrices that make smarter models fit in smaller boxes, needing way less RAM to achieve the same performance

If we quantized it as aggressively as we would have without the distribution/mutation function, the drop in benchmarks would be even more noticeable

It's actually a huge breakthrough and commercially its probably only a short term loss in valuation for the manufacturers

Re: TurboQuant: Redefining AI efficiency with extreme compression

#193

Post by flux3125 »

alkenrinnstet wrote: post #189
parsimo2010 wrote: post #142

This blog post sucks. It does not make me want to read the papers. Look at this figure: https://storage.googleapis.com/gweb-research2023-media/image... The speedup labels on the vertical axis are 0, 2, 2, 4, 6, 8... Why is 2 repeated? Did they just have nano-banana make them some charts? Can they not be bothered to use matplotlib or bokeh and directly render a graph? I don't know, maybe there is some legitimate reaso…

It's slop. The text is also clearly generated by a chatbot with its nonsensical comparisons and bizarrely superlative language.

I bet the paper was vibe written too

Re: TurboQuant: Redefining AI efficiency with extreme compression

#194

Post by mrbonner »

I feel like I’m not the only who feel excited about the whole “compression” tricks while maintaining fidelity in our AI era. In a way, it has a vibe similar to the early 2000s when digital music became popular and the need for lossless compression was paramount. Sort of a pied piper moment for us now . Someone please make a Weisseman score for this stuff.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#195

Post by krackers »

bdcs wrote: post #143

Here's my attempt at a undergrad-level summary (corrections welcome!): The core idea is to quantize KV cache, but do so in a way that destroys minimal information. In this case, it's similarly scores between vectors. The simplest way to do this is to change all the elements from 16bit of precision to, say, 4 bits (Scalar Quant.). These papers improve on it by realizing: almost all the energy (concentration of measure…

Beautiful explanation, thanks!

Re: TurboQuant: Redefining AI efficiency with extreme compression

#196

Post by bdcs »

fc417fc802 wrote: post #179
bdcs wrote: post #146

Earlier quoted context omitted.

Some corrections: the vectors are un-rotated in practice for future query vectors. This could be removed with a slightly different LLM arch. PolarQuant does live on in TurboQuant's codebooks for quantization which borrows from the hyperpolar coords

> added in some gross / highly-pragmatic extra bits for important channels I'm curious what you meant by that. I understood it to only have the MSE quantization vector, a 1-bit QJL vector, and a scalar magnitude. > PolarQuant does live on in TurboQuant's codebooks for quantization which borrows from the hyperpolar coords Isn't the turbo codebook the irregularly spaced centroid grid?

> extra bits per channel

Page 18 of the paper: > As shown in Table 1, our approach outperforms other methods for both Llama-3.1-8B-Instruct and Ministral-7B-Instruct, achieving significantly higher average scores. We evaluate our method using 2.5-bit and 3.5-bit quantization during text generation. These non-integer bit precisions result from our strategy of splitting channels into outlier and non-outlier sets, and applying two independent instances of TurboQuant to each, allocating higher bit precision to outliers. This outlier treatment strategy is consistent with prior work [63, 51] . For example, in our 2.5-bit setup, 32 outlier channels are quantized at 3 bits, while the remaining 96 channels use 2 bits, leading to an effective bit precision of (32 ×3 + 96×2)/128 = 2.5. For 3.5-bit quantization, a different ratio of outliers and regular channels leads to a higher effective bit precision. Despite using fewer bits than competing techniques, TurboQuant maintains performance comparable to unquantized models

So they find channels / indicies-of-the-vector that are important and give them more bits (3 bits) than the rest (2 bits).

>Isn't the turbo codebook the irregularly spaced centroid grid?

yes I believe so. They mention it's informed by the concentration of measure and the uncorrelated/independent vectors after the initial conditioning rotation. I feel like it was informed by PolarQuant, but that may just be how I intuit what's going on (because thinking about this in polar coordinates makes more sense in my head). IOW, I think the irregular spacing is maybe informed by TurboQuant.

However they do say, slightly to the contrary: "We find optimal scalar quantizers for random variables with Beta distributions by solving a continuous 1-dimensional k-means problem using the Max-Lloyd algorithm."

Re: TurboQuant: Redefining AI efficiency with extreme compression

#197

Post by psidium »

photon_lines wrote: post #100
gavinray wrote: post #80

Can someone ELI5 these two concepts please, which make no sense to me: > "TurboQuant starts by randomly rotating the data vectors. This clever step simplifies the data's geometry" I don't understand how taking a series of data and applying a random rotation could mathemetically lead every time to "simpler" geometry. If I throw a bunch of shapes on the ground, tightly packed and touching each other, then rotate all of…

The whole goal of quantisation is to put the data into 'bins' so that it can easily be 'packed' so that you can represent it using less bits (less information). You can think of it like rounding essentially (3.14159 -> 3). Now, sometimes within data, the distribution will be non-ideal for separating it out into bins (let's say that our rounding rules are simple -- we simply use a floor function so 2.45 maps to 2 and…

Wow, thank you for the explanation. Such a complex topic and yet you’ve made it simple to understand.

Re: TurboQuant: Redefining AI efficiency with extreme compression

#198

Post by functional_dev »

psidium wrote: post #197
photon_lines wrote: post #100

Earlier quoted context omitted.

The whole goal of quantisation is to put the data into 'bins' so that it can easily be 'packed' so that you can represent it using less bits (less information). You can think of it like rounding essentially (3.14159 -> 3). Now, sometimes within data, the distribution will be non-ideal for separating it out into bins (let's say that our rounding rules are simple -- we simply use a floor function so 2.45 maps to 2 and…

Wow, thank you for the explanation. Such a complex topic and yet you’ve made it simple to understand.

i wonder what is the limit of quantization when it starts to destroy the logic of weights?

Re: TurboQuant: Redefining AI efficiency with extreme compression

#199

Post by mskkm »

seems to be a scam

"The TurboQuant paper (ICLR 2026) contains serious issues in how it describes RaBitQ, including incorrect technical claims and misleading theory/experiment comparisons. We flagged these issues to the authors before submission. They acknowledged them, but chose not to fix them. The paper was later accepted and widely promoted by Google, reaching tens of millions of views.

We’re speaking up now because once a misleading narrative spreads, it becomes much harder to correct. We’ve written a public comment on openreview (https://openreview.net/forum?id=tO3ASKZlok).

We would greatly appreciate your attention and help in sharing it."

https://x.com/gaoj0017/status/2037532673812443214

Re: TurboQuant: Redefining AI efficiency with extreme compression

#200

Post by ptx »

TeMPOraL wrote: post #97
nvme0n1p1 wrote: post #23

Earlier quoted context omitted.

There are tells all over the page: > Redefining AI efficiency with extreme compression "Redefine" is a favorite word of AI. Honestly no need to read further. > the key-value cache, a high-speed "digital cheat sheet" that stores frequently used information under simple labels No competent engineer would describe a cache as a "cheat sheet". Cheat sheets are static, but caches dynamically update during execution. Studen…

I say you're fixating on the wrong signal here. "Redefine" and "cheat sheet" are normal words people frequently use, and I see worse metaphors in human-written text routinely. It's the structure and rhythm at the sentence and paragraph levels that's the current tell, as SOTA LLMs all seem to overuse clarification constructs like "it's not X, it's Y" and "it's X, an Y and a Z", and "it's X, it's essentially doing Y".…

> they write text better

Not if you view text as a medium for communication, i.e. as a way for a sender to serialize some idea they have in their mind and transfer it to the reader for deserialization.

The AI doesn't know what the sender meant. It can't add any clarity. It can only corrupt and distort whatever message the sender was trying to communicate.

Fixating on these tells is a way for the receiver of the message to detect that it has been corrupted and there is no point in trying to deserialize it. The harder you try to interpret an AI-generated message, the less sense it will make.

Post reply on HN